ML Credit Risk API on Kubernetes

Deploy a credit risk ML API on local Kubernetes with CI/CD and monitoring.

Introduction

30 Second Summary

Every time someone applies for a loan, a machine learning model decides whether they're a safe bet. Getting that model from a notebook into a reliable, scalable API is the core challenge of MLOps.

In this project, you will train a credit risk model, wrap it in a prediction API, deploy it to a local Kubernetes cluster, and automate the entire build-and-test cycle with CI/CD.

What You'll Build

A production-style ML prediction system running entirely on your local machine that scores loan applications for credit risk using a trained XGBoost model served through FastAPI on Kubernetes.

By the end of this project, you'll have:

  • A trained XGBoost credit risk model tracked and registered in MLflow's model registry with full experiment history.
  • A containerized FastAPI prediction service deployed on a local Kind Kubernetes cluster with Prometheus metrics instrumentation.
  • A GitHub Actions CI pipeline that runs tests, builds the Docker image, and validates the container on every push.
  • Secret Mission: Wire up Prometheus and Grafana on the Kind cluster to monitor API request latency, throughput, and prediction distribution in real-time dashboards.

Are there any prerequisites?

You need Docker, Python 3.10+, Git, kubectl, and a free GitHub account already set up on your machine. This project also assumes a Linux environment with at least 16GB RAM.

Set Up Kind Cluster and MLflow

Before you can deploy an ML model to Kubernetes, you need a cluster to deploy to and a way to track your model experiments.

In this step, you will create a local Kubernetes cluster using Kind and start an MLflow tracking server to record your model training runs.

In this step, get ready to:
  • Create a Kind cluster with port mappings for accessing deployed services.
  • Start an MLflow tracking server with a local SQLite backend.
  • Verify both the cluster and MLflow UI are operational.
Create the Kind cluster configuration

Kind needs a configuration file to know which ports from the cluster containers should be accessible on your host machine. Without these port mappings, you would not be able to reach services deployed inside the cluster from your browser or terminal.

  • Create a file called kind-config.yaml in your project root by running:
cat <<EOF > kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    extraPortMappings:
      - containerPort: 30080
        hostPort: 30080
        protocol: TCP
      - containerPort: 30090
        hostPort: 30090
        protocol: TCP
      - containerPort: 30030
        hostPort: 30030
        protocol: TCP
EOF

What does this configuration do?

This YAML file tells Kind to create a single control-plane node with three port mappings.

  • Port 30080 will expose your prediction API later.
  • Port 30090 is reserved for Prometheus (Secret Mission).
  • Port 30030 is reserved for Grafana (Secret Mission).

These are NodePort ranges that Kubernetes uses to expose services outside the cluster.

  • Confirm the file was created by running:
ls kind-config.yaml

You should see kind-config.yaml listed in the output.

  • Create your Kind cluster named credit-risk using the config file by running:
kind create cluster --name credit-risk --config kind-config.yaml

What does this command do?

This spins up a Docker container that acts as a full Kubernetes node. Kind uses kubeadm internally, the same tool used to bootstrap production clusters.

The --name flag gives the cluster a custom name so you can manage multiple clusters if needed. The --config flag points to your port-mapping configuration.

  • Verify the cluster is running and the node is healthy by running:
kubectl get nodes

You should see output showing a single node with Ready in the STATUS column, similar to:

NAME                        STATUS   ROLES           AGE   VERSION
credit-risk-control-plane   Ready    control-plane   30s   v1.x.x

Node showing NotReady?

Wait 30 seconds and run kubectl get nodes again. The node needs a moment to finish bootstrapping its networking layer.

If the node stays NotReady after a minute, check that Docker is running with docker ps. Kind requires Docker to be active.

Help me debug my Kind cluster

✔️ Awesome, I've got everything!

Great. Double check you have saved kind-config.yaml and your node shows Ready.

ⓧ I'd like to double check the full code

Here is the complete kind-config.yaml file:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    extraPortMappings:
      - containerPort: 30080
        hostPort: 30080
        protocol: TCP
      - containerPort: 30090
        hostPort: 30090
        protocol: TCP
      - containerPort: 30030
        hostPort: 30030
        protocol: TCP
Start the MLflow tracking server

MLflow is an experiment tracking platform that logs your model parameters, metrics, and artifacts. You will use it to register the trained model so your API can load it later.

  • Start the MLflow server with a local SQLite database by running:
mlflow server --backend-store-uri sqlite:///mlflow.db --port 5000 &

What does this command do?

  • --backend-store-uri sqlite:///mlflow.db tells MLflow to store experiment data in a local SQLite file called mlflow.db in your current directory.
  • --port 5000 makes the UI and API available at port 5000.
  • The trailing & runs the process in the background so you can continue using the same terminal.
  • Wait a few seconds for the server to start, then verify it is running:
curl http://localhost:5000/health

You should see OK printed in your terminal, confirming the server is responding.

  • Open your browser and navigate to http://localhost:5000.

You should see the MLflow UI with an empty experiments list. This is where your training runs will appear after the next step.

Connection refused on port 5000?

The server may need a few more seconds to initialize. Wait 5 seconds and try the curl command again.

If it still fails, check whether another process is using port 5000 by running lsof -i :5000. If so, stop that process first, then restart the MLflow server.

Help me troubleshoot MLflow connection issues

Your local infrastructure is ready. Next up, you will train an XGBoost credit risk model and register it in MLflow's model registry.

Train and Register the Credit Risk Model

Your cluster is running and MLflow is tracking experiments. Now it is time to train the machine learning model that will power the prediction API.

Credit risk scoring is a classic binary classification problem. Given features about a loan applicant, the model predicts whether they are high-risk or low-risk. You will train an XGBoost classifier on synthetic data, log the experiment to MLflow, and register the resulting model so that later services can load it by name.

In this step, get ready to:
  • Create a training script that generates synthetic credit data and trains an XGBoost model.
  • Run the script and register the champion model in MLflow's Model Registry.
  • Export and validate the model artifact for containerized serving.
Create the training script

The training script generates a synthetic dataset that mimics credit risk features, trains a classifier, and logs everything to MLflow in a single run.

  • Create a file called train.py in the root of your project folder by running:
touch train.py
  • Open train.py in your editor and add the following imports, MLflow configuration, and data generation code:
import mlflow
import mlflow.xgboost
import xgboost as xgb
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, f1_score, roc_auc_score

# Point MLflow at the local tracking server
mlflow.set_tracking_uri("http://localhost:5000")
mlflow.set_experiment("credit-risk")

# Generate synthetic credit risk data
X, y = make_classification(
    n_samples=5000,
    n_features=5,
    n_informative=4,
    n_redundant=1,
    random_state=42,
    weights=[0.7, 0.3],
)

# Name the features for interpretability
feature_names = [
    "income",
    "debt_ratio",
    "credit_history_months",
    "num_accounts",
    "payment_history_score",
]

# Split into training and test sets
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

What does this code do?

This section sets up the experiment infrastructure. mlflow.set_tracking_uri connects to your local MLflow server. mlflow.set_experiment creates (or reuses) an experiment named credit-risk to group all related runs.

The make_classification function from scikit-learn generates 5000 samples with 5 features that simulate real credit application data. The 70/30 class imbalance (weights=[0.7, 0.3]) reflects how most applicants are low-risk in the real world.

  • Below the train/test split, add the MLflow training run that logs parameters, trains the model, evaluates metrics, and registers the model:
with mlflow.start_run(run_name="xgboost-credit-risk"):
    # Define hyperparameters
    params = {
        "n_estimators": 100,
        "max_depth": 5,
        "learning_rate": 0.1,
        "objective": "binary:logistic",
        "eval_metric": "logloss",
        "random_state": 42,
    }
    mlflow.log_params(params)

    # Train the XGBoost classifier
    model = xgb.XGBClassifier(**params)
    model.fit(X_train, y_train)

    # Evaluate on the test set
    y_pred = model.predict(X_test)
    y_proba = model.predict_proba(X_test)[:, 1]

    accuracy = accuracy_score(y_test, y_pred)
    f1 = f1_score(y_test, y_pred)
    auc = roc_auc_score(y_test, y_proba)

    # Log metrics to MLflow
    mlflow.log_metrics({"accuracy": accuracy, "f1_score": f1, "auc_roc": auc})

    # Log and register the model in the Model Registry
    mlflow.xgboost.log_model(
        model,
        artifact_path="model",
        input_example=X_test[:1],
        registered_model_name="credit-risk-model",
    )

    print(f"Accuracy: {accuracy:.4f}")
    print(f"F1 Score: {f1:.4f}")
    print(f"AUC-ROC: {auc:.4f}")

What does this code do?

The with mlflow.start_run() block creates a tracked experiment run. Everything inside it (parameters, metrics, artifacts) is associated with that single run in the MLflow UI.

The mlflow.xgboost.log_model call saves the trained model as an artifact AND registers it in the Model Registry under the name credit-risk-model. This creates version 1 of that registered model automatically.

  • Save train.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved the file before running it.

ⓧ I'd like to double check the full code

import mlflow
import mlflow.xgboost
import xgboost as xgb
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, f1_score, roc_auc_score

mlflow.set_tracking_uri("http://localhost:5000")
mlflow.set_experiment("credit-risk")

X, y = make_classification(
    n_samples=5000,
    n_features=5,
    n_informative=4,
    n_redundant=1,
    random_state=42,
    weights=[0.7, 0.3],
)

feature_names = [
    "income",
    "debt_ratio",
    "credit_history_months",
    "num_accounts",
    "payment_history_score",
]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

with mlflow.start_run(run_name="xgboost-credit-risk"):
    params = {
        "n_estimators": 100,
        "max_depth": 5,
        "learning_rate": 0.1,
        "objective": "binary:logistic",
        "eval_metric": "logloss",
        "random_state": 42,
    }
    mlflow.log_params(params)

    model = xgb.XGBClassifier(**params)
    model.fit(X_train, y_train)

    y_pred = model.predict(X_test)
    y_proba = model.predict_proba(X_test)[:, 1]

    accuracy = accuracy_score(y_test, y_pred)
    f1 = f1_score(y_test, y_pred)
    auc = roc_auc_score(y_test, y_proba)

    mlflow.log_metrics({"accuracy": accuracy, "f1_score": f1, "auc_roc": auc})

    mlflow.xgboost.log_model(
        model,
        artifact_path="model",
        input_example=X_test[:1],
        registered_model_name="credit-risk-model",
    )

    print(f"Accuracy: {accuracy:.4f}")
    print(f"F1 Score: {f1:.4f}")
    print(f"AUC-ROC: {auc:.4f}")
Run the training script and register the champion model

With the training script saved, you can now execute it. The script trains the model, logs everything to MLflow, and registers the model in one go.

  • Run the training script by executing:
python train.py
  • You should see output showing the model's accuracy, F1 score, and AUC-ROC values (all above 0.85).
  • Open the MLflow UI at http://localhost:5000 in your browser.
  • Click on the credit-risk experiment in the left sidebar.
  • You should see a completed run named xgboost-credit-risk with logged parameters and metrics.

What is the MLflow Model Registry?

The Model Registry is MLflow's central catalog for production models. When you logged the model with registered_model_name="credit-risk-model", MLflow created version 1 of a registered model. You can assign aliases (like "champion") to specific versions so downstream services always load the correct model without hardcoding version numbers.

Now assign the "champion" alias to version 1 so that downstream services can reference it by alias instead of version number.

  • Set the champion alias by running this Python command:
python -c "
import mlflow
from mlflow import MlflowClient
mlflow.set_tracking_uri('http://localhost:5000')
client = MlflowClient()
client.set_registered_model_alias('credit-risk-model', 'champion', '1')
print('Champion alias set for credit-risk-model version 1')
"
  • You should see the confirmation message printed in your terminal.

Seeing a connection error?

Make sure your MLflow server is still running in another terminal tab. If it stopped, restart it with mlflow server --backend-store-uri sqlite:///mlflow.db --port 5000.

If you see an error about the model not being found, confirm that train.py completed without errors and that you can see the run in the MLflow UI.

help me debug my MLflow connection error

Validate the model and export it for serving

Before containerizing the API, you need to export the model as a standalone binary that loads without the full MLflow stack. This keeps your Docker image lean and removes the MLflow runtime dependency from production.

  • Validate the registered model loads correctly and export it to a local directory by running:
python -c "
import mlflow
import mlflow.xgboost
import numpy as np
import os

mlflow.set_tracking_uri('http://localhost:5000')

# Load the champion model from the registry
model = mlflow.xgboost.load_model('models:/credit-risk-model@champion')
print('Model loaded successfully from registry')

# Test prediction with sample data
test_input = np.array([[0.5, -0.3, 1.2, 0.8, -0.5]])
prediction = model.predict(test_input)
proba = model.predict_proba(test_input)
print(f'Prediction: {prediction[0]}, Probability: {proba[0][1]:.4f}')

# Export the native model for containerized serving
os.makedirs('model', exist_ok=True)
model.save_model('model/model.ubj')
print('Model exported to model/model.ubj')
"

Why export the model separately?

The MLflow registry stores models with metadata, environment specs, and multiple serialization layers. Loading from the registry requires the full MLflow SDK. By exporting the native .ubj file (Universal Binary JSON), the prediction API can load the model with just the xgboost package. This makes the Docker image smaller and faster to start.

  • Confirm the model file exists by running:
ls model/
  • You should see model.ubj in the output.

Model export failed?

If you see an error about the model not being found in the registry, double check that you ran the alias command in the previous substep. The URI models:/credit-risk-model@champion requires the alias to exist.

If the prediction test fails with a shape error, ensure your test input has exactly 5 features (matching the training data).

help me debug my model export from MLflow

Your model is trained, registered, aliased, and exported. Next up, you will build the FastAPI prediction service that loads this model and exposes it as a REST endpoint.

Build the FastAPI Prediction Service

You have a trained model registered in MLflow and exported as a native .ubj file. Now you need a way for other systems to actually use that model.

In this step, you will build a FastAPI prediction service that loads the XGBoost model and exposes it as an HTTP endpoint. You will also instrument the service with Prometheus metrics so you can monitor prediction latency and throughput once it is deployed to Kubernetes.

In this step, get ready to:
  • Create a FastAPI app with Pydantic request/response schemas and a model-loading startup hook.
  • Instrument the prediction endpoint with Prometheus counters and histograms.
  • Write pytest tests to validate the API endpoints work correctly.
Create the app directory and API foundation

Every Python package needs an __init__.py file to be importable. You will create the app/ directory first, then build the FastAPI application file inside it.

  • Create the app/ directory and an empty __init__.py file by running these commands:
mkdir -p app
touch app/__init__.py
  • Confirm the directory was created by running:
ls app/

You should see __init__.py listed in the output.

  • Create a new file called app/main.py and add the following imports and application setup:
import os
import time
import numpy as np
import xgboost as xgb
from fastapi import FastAPI
from pydantic import BaseModel
from prometheus_client import (
    Counter,
    Histogram,
    generate_latest,
    CONTENT_TYPE_LATEST,
)
from fastapi.responses import Response

app = FastAPI(title="Credit Risk Prediction API", version="1.0.0")

MODEL_PATH = os.getenv("MODEL_PATH", "model/model.ubj")

model = None


class PredictionRequest(BaseModel):
    income: float
    debt_ratio: float
    credit_history_months: float
    num_accounts: float
    payment_history_score: float


class PredictionResponse(BaseModel):
    risk_score: float
    decision: str

What does this code do?

This sets up the skeleton of the prediction API.

  • FastAPI creates the web application with automatic OpenAPI documentation.
  • MODEL_PATH reads from an environment variable (defaulting to model/model.ubj), so you can override the path when containerizing later.
  • PredictionRequest defines exactly what fields the API expects. Pydantic validates incoming JSON automatically.
  • PredictionResponse defines the shape of what the API returns: a risk score and a binary decision string.
Add Prometheus metrics and prediction endpoints

With the schemas defined, you now need the actual logic that loads the model, handles predictions, and tracks performance. Prometheus metrics let you measure request latency and prediction distribution without modifying your core logic.

  • Add the Prometheus metric definitions below the PredictionResponse class in app/main.py:
REQUEST_LATENCY = Histogram(
    "prediction_request_duration_seconds",
    "Time spent processing prediction request",
)
PREDICTION_COUNT = Counter(
    "predictions_total",
    "Total number of predictions made",
)
PREDICTION_CLASS = Counter(
    "prediction_class_total",
    "Predictions by class",
    ["predicted_class"],
)

What are these metrics?

  • REQUEST_LATENCY is a Histogram that tracks how long each prediction request takes in seconds. Histograms give you percentile breakdowns (p50, p95, p99).
  • PREDICTION_COUNT is a Counter that increments every time a prediction is made. Useful for throughput monitoring.
  • PREDICTION_CLASS is a Counter partitioned by the label predicted_class. This shows you the distribution of high-risk vs low-risk predictions over time.
  • Add the startup hook and all three endpoints below the metric definitions in app/main.py:
@app.on_event("startup")
def load_model():
    global model
    model = xgb.XGBClassifier()
    model.load_model(MODEL_PATH)


@app.get("/health")
def health():
    return {"status": "healthy", "model_loaded": model is not None}


@app.post("/predict", response_model=PredictionResponse)
def predict(request: PredictionRequest):
    start_time = time.time()

    features = np.array(
        [[
            request.income,
            request.debt_ratio,
            request.credit_history_months,
            request.num_accounts,
            request.payment_history_score,
        ]]
    )

    proba = model.predict_proba(features)[0][1]
    decision = "high_risk" if proba >= 0.5 else "low_risk"

    PREDICTION_COUNT.inc()
    PREDICTION_CLASS.labels(predicted_class=decision).inc()

    duration = time.time() - start_time
    REQUEST_LATENCY.observe(duration)

    return PredictionResponse(risk_score=round(float(proba), 4), decision=decision)


@app.get("/metrics")
def metrics():
    return Response(content=generate_latest(), media_type=CONTENT_TYPE_LATEST)

What do these endpoints do?

  • load_model() runs once when the server starts. It loads the XGBoost model file from disk so it is ready before any requests arrive.
  • GET /health returns a simple JSON check. Kubernetes will use this as a liveness and readiness probe.
  • POST /predict takes a loan application as JSON, runs it through the model, records Prometheus metrics, and returns the risk score with a decision.
  • GET /metrics exposes all Prometheus metrics in the text format that Prometheus scrapers expect.

Why use @app.on_event instead of lifespan?

FastAPI's official docs mark @app.on_event("startup") as deprecated in favor of the newer lifespan context manager. However, the legacy pattern still works and is simpler for a single startup task like loading a model.

For this project, the simpler pattern keeps the code readable. In production apps with shared resources or cleanup logic, you would migrate to the lifespan approach.

  • Save app/main.py.

✔️ Awesome, I've got everything!

Great. Double check you have saved app/main.py before moving on.

ⓧ I'd like to double check the full code

import os
import time
import numpy as np
import xgboost as xgb
from fastapi import FastAPI
from pydantic import BaseModel
from prometheus_client import (
    Counter,
    Histogram,
    generate_latest,
    CONTENT_TYPE_LATEST,
)
from fastapi.responses import Response

app = FastAPI(title="Credit Risk Prediction API", version="1.0.0")

MODEL_PATH = os.getenv("MODEL_PATH", "model/model.ubj")

model = None


class PredictionRequest(BaseModel):
    income: float
    debt_ratio: float
    credit_history_months: float
    num_accounts: float
    payment_history_score: float


class PredictionResponse(BaseModel):
    risk_score: float
    decision: str


REQUEST_LATENCY = Histogram(
    "prediction_request_duration_seconds",
    "Time spent processing prediction request",
)
PREDICTION_COUNT = Counter(
    "predictions_total",
    "Total number of predictions made",
)
PREDICTION_CLASS = Counter(
    "prediction_class_total",
    "Predictions by class",
    ["predicted_class"],
)


@app.on_event("startup")
def load_model():
    global model
    model = xgb.XGBClassifier()
    model.load_model(MODEL_PATH)


@app.get("/health")
def health():
    return {"status": "healthy", "model_loaded": model is not None}


@app.post("/predict", response_model=PredictionResponse)
def predict(request: PredictionRequest):
    start_time = time.time()

    features = np.array(
        [[
            request.income,
            request.debt_ratio,
            request.credit_history_months,
            request.num_accounts,
            request.payment_history_score,
        ]]
    )

    proba = model.predict_proba(features)[0][1]
    decision = "high_risk" if proba >= 0.5 else "low_risk"

    PREDICTION_COUNT.inc()
    PREDICTION_CLASS.labels(predicted_class=decision).inc()

    duration = time.time() - start_time
    REQUEST_LATENCY.observe(duration)

    return PredictionResponse(risk_score=round(float(proba), 4), decision=decision)


@app.get("/metrics")
def metrics():
    return Response(content=generate_latest(), media_type=CONTENT_TYPE_LATEST)

Import errors when you open the file?

Make sure your virtual environment is still activated. If your editor shows red underlines for prometheus_client or xgboost, confirm the packages are installed in your active environment.

Help me debug import errors in app/main.py

Write tests and verify the API works

Before deploying to Kubernetes, you need confidence that the endpoints behave correctly. Writing automated tests now means you can run them in CI later without manual verification.

  • Create app/test_api.py with the following test suite:
import pytest
from fastapi.testclient import TestClient
from app.main import app, load_model

import os

os.environ["MODEL_PATH"] = "model/model.ubj"


@pytest.fixture(autouse=True, scope="module")
def setup_model():
    load_model()


client = TestClient(app)


def test_health():
    response = client.get("/health")
    assert response.status_code == 200
    data = response.json()
    assert data["status"] == "healthy"
    assert data["model_loaded"] is True


def test_predict():
    payload = {
        "income": 0.5,
        "debt_ratio": -0.3,
        "credit_history_months": 1.2,
        "num_accounts": 0.8,
        "payment_history_score": -0.5,
    }
    response = client.post("/predict", json=payload)
    assert response.status_code == 200
    data = response.json()
    assert "risk_score" in data
    assert "decision" in data
    assert 0.0 <= data["risk_score"] <= 1.0
    assert data["decision"] in ["high_risk", "low_risk"]


def test_metrics():
    response = client.get("/metrics")
    assert response.status_code == 200
    assert "predictions_total" in response.text

What do these tests verify?

  • test_health() confirms the health endpoint returns 200 and reports the model is loaded.
  • test_predict() sends a sample loan application and verifies the response contains a valid risk score between 0 and 1, plus a decision string.
  • test_metrics() confirms the Prometheus metrics endpoint returns the predictions_total counter (proving instrumentation is working).
  • The setup_model() fixture calls load_model() directly so the model is loaded before any test runs, without relying on the startup event.
  • Save app/test_api.py.
  • Run the test suite by running this command:
pytest app/test_api.py -v

You should see all three tests pass with green output.

Tests failing with ModuleNotFoundError?

Make sure you are running pytest from your project root directory (the folder that contains both app/ and model/). If you are inside the app/ folder, Python cannot resolve the import path correctly.

Also confirm pytest is installed in your virtual environment. If not, install it with pip install pytest.

Help me fix failing pytest tests

✔️ Awesome, I've got everything!

Great. Double check you have saved app/test_api.py and all three tests are passing.

ⓧ I'd like to double check the full code

import pytest
from fastapi.testclient import TestClient
from app.main import app, load_model

import os

os.environ["MODEL_PATH"] = "model/model.ubj"


@pytest.fixture(autouse=True, scope="module")
def setup_model():
    load_model()


client = TestClient(app)


def test_health():
    response = client.get("/health")
    assert response.status_code == 200
    data = response.json()
    assert data["status"] == "healthy"
    assert data["model_loaded"] is True


def test_predict():
    payload = {
        "income": 0.5,
        "debt_ratio": -0.3,
        "credit_history_months": 1.2,
        "num_accounts": 0.8,
        "payment_history_score": -0.5,
    }
    response = client.post("/predict", json=payload)
    assert response.status_code == 200
    data = response.json()
    assert "risk_score" in data
    assert "decision" in data
    assert 0.0 <= data["risk_score"] <= 1.0
    assert data["decision"] in ["high_risk", "low_risk"]


def test_metrics():
    response = client.get("/metrics")
    assert response.status_code == 200
    assert "predictions_total" in response.text

Your prediction service is tested and ready. Next up, you will containerize it with Docker and deploy it to your Kind Kubernetes cluster.

Containerize and Deploy to Kind

Your FastAPI service passes all tests locally. The next challenge is packaging it so Kubernetes can run it as a managed workload with replicas, health checks, and automatic restarts.

In this step, you will build a Docker image containing the API and model, then deploy it to your Kind cluster using Kubernetes manifests.

In this step, get ready to:
  • Build a Docker image that bundles the prediction service and trained model.
  • Deploy the image to Kind with a Deployment and NodePort Service.
  • Verify the live API responds to prediction requests from your host machine.
Write the Dockerfile and build the image

Docker needs a requirements.txt to know which Python packages to install inside the container. This file pins the exact versions your API depends on.

  • Create a file called requirements.txt in your project root with the following contents:
fastapi==0.136.3
uvicorn==0.34.0
xgboost==3.2.0
numpy<3
scikit-learn<2
prometheus-client==0.25.0
pydantic>=2.0

What does this file do?

Each line pins a dependency to a specific version (or version range). Pinning ensures the container installs the same library versions you tested locally. Without pins, a future build might pull a breaking update.

  • Save requirements.txt.

✔️ Awesome, I've got everything!

Great. Make sure the file is saved in your project root (the same level as train.py).

ⓧ I'd like to double check the full code

fastapi==0.136.3
uvicorn==0.34.0
xgboost==3.2.0
numpy<3
scikit-learn<2
prometheus-client==0.25.0
pydantic>=2.0

Now you need a Dockerfile that tells Docker how to assemble the image. The image must contain your Python dependencies, the app/ code, and the exported model/ artifact.

  • Create a file called Dockerfile (no extension) in your project root with the following contents:
FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app/ ./app/
COPY model/ ./model/

EXPOSE 8000

CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]

What does this Dockerfile do?

  • FROM python:3.11-slim starts from a lightweight Python base image.
  • COPY requirements.txt and RUN pip install install dependencies first. Docker caches this layer, so rebuilds are fast unless dependencies change.
  • COPY app/ and COPY model/ add your application code and the trained XGBoost model binary.
  • CMD runs uvicorn on port 8000 when the container starts.
  • Save Dockerfile.

Seeing a 'file not found' error on save?

Make sure the file is named exactly Dockerfile with a capital D and no file extension. Some editors add .txt automatically.

Help me troubleshoot my Dockerfile name and location.

✔️ Awesome, I've got everything!

Great. Double check you have saved the file in your project root alongside requirements.txt.

ⓧ I'd like to double check the full code

FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app/ ./app/
COPY model/ ./model/

EXPOSE 8000

CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]

With both files in place, you can build the Docker image and load it into your Kind cluster.

  • Build the Docker image by running this command:
docker build -t credit-risk-api:latest .

What does this command do?

The -t credit-risk-api:latest flag tags the image with a name and version. The . at the end tells Docker to use the current directory as the build context (where it looks for your Dockerfile and files to copy).

  • You should see Docker pulling the base image, installing packages, and finishing with a success message.
  • Load the built image into your Kind cluster by running:
kind load docker-image credit-risk-api:latest --name credit-risk

Why load the image into Kind?

Kind runs Kubernetes inside Docker containers. Those containers have their own image store, separate from your host's Docker daemon. Loading the image copies it from your local Docker into the Kind node so Kubernetes can find it.

  • You should see output confirming the image was loaded onto the Kind node.

Build failed with 'no such file or directory'?

Make sure you are running the docker build command from your project root directory (the folder containing Dockerfile). Confirm model/model.ubj exists by running ls model/.

Help me debug my Docker build.

Write Kubernetes manifests and deploy

Kubernetes uses YAML manifests to declare what should run and how it should be exposed. You need two resources: a Deployment (runs your pods) and a Service (routes traffic to them).

  • Create the k8s/ directory by running:
mkdir k8s
  • Create a file called k8s/deployment.yaml with the following contents:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: credit-risk-api
  labels:
    app: credit-risk-api
spec:
  replicas: 2
  selector:
    matchLabels:
      app: credit-risk-api
  template:
    metadata:
      labels:
        app: credit-risk-api
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8000"
        prometheus.io/path: "/metrics"
    spec:
      containers:
        - name: credit-risk-api
          image: credit-risk-api:latest
          imagePullPolicy: Never
          ports:
            - containerPort: 8000
          resources:
            requests:
              memory: "256Mi"
              cpu: "250m"
            limits:
              memory: "512Mi"
              cpu: "500m"
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 10
            periodSeconds: 30
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 5
            periodSeconds: 10

What does this manifest define?

  • replicas: 2 runs two copies of the API for availability.
  • imagePullPolicy: Never tells Kubernetes to use the local image you loaded (never pull from a registry).
  • The resources block sets memory and CPU guardrails so one pod cannot starve the node.
  • livenessProbe and readinessProbe hit /health to confirm the container is alive and ready to serve traffic.
  • The prometheus.io annotations tell Prometheus where to scrape metrics (used in the Secret Mission).
  • Save k8s/deployment.yaml.

✔️ Awesome, I've got everything!

Great. Make sure you saved the file inside the k8s/ directory.

ⓧ I'd like to double check the full code

apiVersion: apps/v1
kind: Deployment
metadata:
  name: credit-risk-api
  labels:
    app: credit-risk-api
spec:
  replicas: 2
  selector:
    matchLabels:
      app: credit-risk-api
  template:
    metadata:
      labels:
        app: credit-risk-api
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8000"
        prometheus.io/path: "/metrics"
    spec:
      containers:
        - name: credit-risk-api
          image: credit-risk-api:latest
          imagePullPolicy: Never
          ports:
            - containerPort: 8000
          resources:
            requests:
              memory: "256Mi"
              cpu: "250m"
            limits:
              memory: "512Mi"
              cpu: "500m"
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 10
            periodSeconds: 30
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 5
            periodSeconds: 10

Next, define a Service that routes external traffic to your pods.

  • Create a file called k8s/service.yaml with the following contents:
apiVersion: v1
kind: Service
metadata:
  name: credit-risk-api
  labels:
    app: credit-risk-api
spec:
  type: NodePort
  selector:
    app: credit-risk-api
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8000
      nodePort: 30080

What does this Service do?

A NodePort Service opens a fixed port (30080) on every cluster node. Since your kind-config.yaml maps container port 30080 to host port 30080, traffic from localhost:30080 reaches the pods inside Kind.

The Service forwards port 80 to targetPort: 8000 inside each pod, which is where uvicorn listens.

  • Save k8s/service.yaml.

✔️ Awesome, I've got everything!

Great. You now have both manifests ready in the k8s/ directory.

ⓧ I'd like to double check the full code

apiVersion: v1
kind: Service
metadata:
  name: credit-risk-api
  labels:
    app: credit-risk-api
spec:
  type: NodePort
  selector:
    app: credit-risk-api
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8000
      nodePort: 30080
  • Apply both manifests to your cluster by running:
kubectl apply -f k8s/
  • You should see output confirming both the Deployment and Service were created.

Seeing 'error: no objects passed to apply'?

Make sure you are running the command from the project root (one level above the k8s/ directory). Check that both .yaml files exist inside k8s/ by running ls k8s/.

Help me debug my kubectl apply command.

Verify the deployment end-to-end

The Deployment creates pods, and Kubernetes rolls them out over a few seconds. You need to confirm the pods reach a Running state before sending traffic.

  • Check the pod status by running:
kubectl get pods
  • Wait until both pods show 1/1 in the READY column and Running in the STATUS column. This may take 10-20 seconds while the containers start and pass the readiness probe.
  • If the pods are not yet ready, re-run kubectl get pods after a few seconds.

Pods stuck in CrashLoopBackOff?

Check the pod logs with kubectl logs <pod-name> (replace <pod-name> with the name from kubectl get pods). Common causes:

  • The model file is missing inside the container. Rebuild the image from the correct directory.
  • A dependency failed to install. Check the build output for pip errors.
  • The image was not loaded into Kind. Re-run the kind load docker-image command.

Help me debug my CrashLoopBackOff pods.

  • Once both pods are Running, test the live API by running:
curl -X POST http://localhost:30080/predict \
  -H "Content-Type: application/json" \
  -d '{"income": 0.5, "debt_ratio": -0.3, "credit_history_months": 1.2, "num_accounts": 0.8, "payment_history_score": -0.5}'
  • You should see a JSON response containing a risk_score (a decimal between 0 and 1) and a decision (either high_risk or low_risk).

What just happened?

Your curl request hit localhost:30080, which the Kind port mapping forwarded into the cluster. The NodePort Service routed it to one of the two pods, which ran the XGBoost model and returned a prediction. This is the same flow a production load balancer would use.

Getting 'connection refused' on localhost:30080?

Confirm the Service is active by running kubectl get svc. Verify that nodePort: 30080 appears in the output. Also check that your Kind cluster was created with the kind-config.yaml that maps port 30080.

Help me debug my Kind port mapping.

Your ML model is now running on Kubernetes with health monitoring and automatic traffic routing. Next up, you will automate this build-and-test workflow with a GitHub Actions CI pipeline.

Set Up the GitHub Actions CI Pipeline

Your credit risk API is running on Kubernetes with pods serving predictions at localhost:30080. The next step is making sure every code change is automatically tested and validated before it reaches your cluster.

CI/CD (Continuous Integration / Continuous Delivery) automates the "did I break anything?" question. In this step, you will create a GitHub Actions workflow that runs your test suite and builds a Docker image on every push to main.

In this step, get ready to:
  • Create a GitHub Actions workflow file with a test job that installs dependencies and runs pytest.
  • Add a build job that builds the Docker image and smoke tests the container.
  • Push the project to GitHub and verify both jobs pass with green checkmarks.
Create the CI workflow with the test job

GitHub Actions looks for workflow files inside a .github/workflows/ directory at the root of your repository. Each YAML file in that directory defines a separate pipeline.

  • Create the workflow directory by running this command:
mkdir -p .github/workflows
  • Confirm the directory exists by running:
ls -la .github/workflows

You should see an empty directory listed.

  • Create the workflow file with the test job by adding the following content to .github/workflows/ci.yml:
name: CI Pipeline

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: Install dependencies
        run: |
          pip install -r requirements.txt
          pip install pytest

      - name: Run tests
        run: pytest app/test_api.py -v

What does this workflow do?

The on: block triggers the pipeline on every push to main and on pull requests targeting main.

  • actions/checkout@v4 clones your repository into the runner.
  • actions/setup-python@v5 installs Python 3.11 on the runner.
  • The install step grabs your production dependencies from requirements.txt plus pytest for testing.
  • The final step runs your three tests in verbose mode against app/test_api.py.
  • Save .github/workflows/ci.yml.

File not saving to the right location?

Make sure the directory path is exactly .github/workflows/ (note the leading dot). GitHub ignores workflow files placed anywhere else.

help me figure out why my workflow file isn't in the right place

Add the Docker build and smoke test job

The test job validates your code logic. A second job will validate that the code also builds into a working container. This catches Dockerfile errors and missing dependencies before they reach your cluster.

  • In .github/workflows/ci.yml, add the following build job below the test job (at the same indentation level as test:):
  build:
    runs-on: ubuntu-latest
    needs: test
    steps:
      - uses: actions/checkout@v4

      - name: Build Docker image
        run: docker build -t credit-risk-api:${{ github.sha }} .

      - name: Smoke test container
        run: |
          docker run -d --name test-api -p 8000:8000 credit-risk-api:${{ github.sha }}
          sleep 5
          curl --fail http://localhost:8000/health
          docker stop test-api

What does the build job do?

The needs: test line means this job only runs after the test job passes. If tests fail, the build never starts.

  • It tags the Docker image with the Git commit SHA (${{ github.sha }}) so every build is uniquely traceable.
  • The smoke test starts the container, waits 5 seconds for it to boot, then hits /health with curl --fail. If the endpoint returns anything other than HTTP 200, the job fails.
  • Save .github/workflows/ci.yml.

YAML indentation errors?

YAML is sensitive to spaces. The build: key must align with test: (both indented 2 spaces under jobs:). Mixing tabs and spaces will cause parse failures.

help me fix my YAML indentation in ci.yml

✔️ Awesome, I've got everything!

Great. Make sure you have saved .github/workflows/ci.yml with both the test and build jobs before moving on.

ⓧ I'd like to double check the full code

Here is the complete .github/workflows/ci.yml file:

name: CI Pipeline

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: Install dependencies
        run: |
          pip install -r requirements.txt
          pip install pytest

      - name: Run tests
        run: pytest app/test_api.py -v

  build:
    runs-on: ubuntu-latest
    needs: test
    steps:
      - uses: actions/checkout@v4

      - name: Build Docker image
        run: docker build -t credit-risk-api:${{ github.sha }} .

      - name: Smoke test container
        run: |
          docker run -d --name test-api -p 8000:8000 credit-risk-api:${{ github.sha }}
          sleep 5
          curl --fail http://localhost:8000/health
          docker stop test-api
Push to GitHub and verify the pipeline

The workflow only triggers when code lands on GitHub. You need to create a remote repository and push your project to it.

  • Open your browser and go to github.com.
  • Click the + icon in the top-right corner, then select New repository.
  • Enter credit-risk-api as the repository name.
  • Leave the repository set to Public.
  • Do not check any boxes for README, .gitignore, or license.
  • Click Create repository.

GitHub shows a quick setup page with the remote URL. Copy the HTTPS URL shown on that page.

  • Return to your terminal and stage all project files by running:
git add app/ model/ k8s/ .github/ train.py requirements.txt Dockerfile kind-config.yaml
  • Commit the files by running:
git commit -m "Add credit risk API with CI pipeline"
  • Ensure your default branch is named main by running:
git branch -M main
  • Add your GitHub repository as the remote origin by running:
git remote add origin [[YOUR_REPO_URL="https://github.com/YOUR_USERNAME/credit-risk-api.git"]]
  • Push your code to GitHub by running:
git push -u origin main

What happens when you push?

The moment your code hits the main branch on GitHub, the on: push trigger fires. GitHub spins up an Ubuntu runner, checks out your code, and starts executing the test job. Once that passes, the build job kicks off.

  • In your browser, navigate to your repository on GitHub.
  • Click the Actions tab at the top of the repository page.
  • Click into the running workflow (it will be named Add credit risk API with CI pipeline from your commit message).
  • Wait for both the test and build jobs to complete. You should see green checkmarks next to both.

Click on either job to expand its logs and see the individual step outputs.

Seeing a red X on the test job?

Click the failed job to expand the logs. Common causes:

  • Missing model/model.ubj in the commit. The tests load the model file at startup. Make sure git add model/ included the binary.
  • The app/__init__.py file was not committed. Without it, Python cannot import the app package. Run git add app/__init__.py, commit, and push again.
  • YAML syntax error in ci.yml. Check that indentation uses spaces (not tabs) and the run: | blocks are indented correctly.

help me debug my failing GitHub Actions workflow

Secret mission

Prometheus + Grafana Monitoring Stack

Your API is running, but can you see what it's doing? Deploy a full observability stack to your Kind cluster. Install Prometheus and Grafana via Helm, wire up scrape targets to your prediction service, and build real-time dashboards showing request rate, p95 latency, and prediction class distribution.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally, so there are no ongoing costs.

Resources you used:

  • Kind Kubernetes cluster (local Docker containers).
  • MLflow SQLite database and artifact files (local filesystem).
  • Docker images (local storage).
  • Python virtual environment and project files.

Keep everything running

No action needed. Choose this if you're still actively building or experimenting with your credit risk API.

  • The Kind cluster persists across reboots unless you stop Docker.
  • MLflow data stays in the project directory.
  • Docker images remain cached on your machine.
  • The Kind binary stays at /usr/local/bin/kind.

Pause - I'll come back to this later

Shut down the running cluster containers to free up memory. Your project files and images stay intact so you can restart later.

  • Stop the MLflow server if it is still running. Press Ctrl+C in the terminal where it is running.
  • Stop the Kind cluster container by running:
docker stop credit-risk-control-plane
  • When you want to resume, start the container again by running:
docker start credit-risk-control-plane
  • Your project files, Docker images, and MLflow data all remain on disk ready to use.

Delete - I don't want to use this again

Remove all project resources and start fresh. This frees up disk space and removes all Docker containers associated with the cluster.

  • Stop the MLflow server if it is still running. Press Ctrl+C in the terminal where it is running.
  • Delete the Kind cluster by running:
kind delete cluster --name credit-risk
  • Remove the Docker image by running:
docker rmi credit-risk-api:latest
  • Remove the Kind binary by running:
sudo rm /usr/local/bin/kind
  • Delete the project directory and all its contents by running:
cd ~/Desktop
rm -rf credit-risk-api
  • This removes your training script, model artifacts, MLflow database, Kubernetes manifests, app code, and virtual environment.

Nice Work!

Nice Work!

You just built a production-grade ML serving pipeline from scratch. A trained credit risk model now runs on Kubernetes, served by a FastAPI endpoint, validated by automated CI, and monitored with real-time observability.

You've learned how to:

  • Train an XGBoost credit risk model and track experiments with MLflow (model registry, versioning, and metrics).
  • Build a production-ready FastAPI prediction service with Prometheus metrics instrumentation, containerize it with Docker, and deploy it to a local Kind Kubernetes cluster.
  • Create a GitHub Actions CI pipeline that runs tests and builds the container on every push.
  • (Secret Mission) Deploy a full Prometheus + Grafana observability stack and build real-time monitoring dashboards for the ML API.

Ready to quiz yourself?