ML Monitoring with Prometheus and Grafana
Build a production ML monitoring stack with drift detection and alerting.
Introduction
30 Second Summary
Machine learning models don't just break with error messages. They silently degrade as the real-world data they receive drifts away from what they were trained on.
In this project, you will build a production-grade ML monitoring stack that instruments a model inference API with custom metrics, detects data drift in real time, and triggers severity-based alerts through a full observability pipeline.
What You'll Build
A local MLOps monitoring stack using FastAPI, Prometheus, Grafana, and Alertmanager deployed via Docker Compose.
By the end of this project, you'll have:
- A production FastAPI inference service instrumented with 7 custom Prometheus metrics covering latency, throughput, errors, drift, confidence, and feature statistics.
- A 4-service observability stack deployed via Docker Compose with recording rules, 6 alert rules across 3 severity groups, and an 8-panel Grafana dashboard provisioned as code.
- A working alerting pipeline where drifted model inputs trigger severity-based alerts routed through Alertmanager with inhibition rules, grouping, and webhook delivery.
- Secret Mission: Add model version tracking and build a Grafana comparison panel showing performance differences across model versions.
Are there any prerequisites?
You should be comfortable with Python and have basic familiarity with Docker containers.
Prior exposure to Prometheus or Grafana is helpful but not required. The project walks through every configuration file from scratch.
Before We Start
ML models in production can silently degrade over time. Data drift, latency spikes, and confidence drops all happen without warning unless you have a monitoring stack watching for them.
In this step, you'll create the project structure, install the Python packages your monitoring service needs, and confirm your local tools are ready to go.
In this step, get ready to:
- Create a production-layout project directory with subdirectories for each service.
- Set up a Python virtual environment and install all dependencies.
- Verify Docker, Docker Compose, and Python are available.
Create the project directory structure
A production ML project separates concerns into distinct directories. Each service in the stack gets its own configuration folder so that Docker Compose can mount them independently.
- Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
- Type Terminal and press Enter to open it.
- Move to your Desktop by running this command:
cd ~/Desktop- Create your project folder and move into it by running these commands:
mkdir ml-monitoring
cd ml-monitoring- Create the subdirectories for each service by running this command:
mkdir -p app model prometheus grafana/provisioning/datasources grafana/provisioning/dashboards grafana/dashboards alertmanager scriptsWhat do these directories hold?
- app/ holds your FastAPI inference service code.
- model/ stores your trained model artifact and reference data.
- prometheus/ holds Prometheus scrape config and alert rules.
- grafana/ contains Grafana provisioning files and dashboard JSON.
- alertmanager/ stores the Alertmanager routing and receiver configuration.
- scripts/ is where your traffic generator script will live.
- Create an empty __init__.py file so Python treats app/ as a package:
touch app/__init__.py- Confirm your directory structure was created by running:
lsYou should see alertmanager, app, grafana, model, prometheus, and scripts listed in the output.
Missing directories?
Make sure you ran the mkdir -p command from inside the ml-monitoring folder. If you see only your Desktop contents, run cd ml-monitoring first, then re-run the mkdir command.
Need more help? Help me troubleshoot my missing directories in ml-monitoring.
Set up a Python virtual environment and install dependencies
A virtual environment keeps this project's packages isolated from your system Python. This prevents version conflicts with other projects on your machine.
- Create a virtual environment by running this command:
python3 -m venv venv- Activate the virtual environment:
macOS
source venv/bin/activateWindows
venv\Scripts\activateYou should see (venv) appear at the beginning of your terminal prompt.
Don't see (venv) in your prompt?
On macOS, make sure you typed source before the path. On Windows, if you get an execution policy error, run Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy RemoteSigned in PowerShell first, then try the activate command again.
Still stuck? Help me activate my Python virtual environment.
Now you need a requirements.txt file that pins exact package versions. This ensures your Docker image installs the same versions as your local environment.
- Open Antigravity IDE, then open the ml-monitoring folder from your Desktop.
- Create a new file called requirements.txt and paste the following content:
fastapi==0.136.3
uvicorn==0.49.0
scikit-learn==1.9.0
prometheus-client==0.25.0
joblib==1.4.2
numpy==2.2.6
scipy==1.15.3
pydantic==2.11.4What are these packages for?
- fastapi and uvicorn serve your ML model as a REST API.
- scikit-learn provides the classifier you'll train and serve.
- prometheus-client exposes custom metrics from your Python service.
- joblib serializes and loads model artifacts from disk.
- numpy and scipy handle numerical computation and statistical tests for drift detection.
- pydantic validates request and response data shapes.
- Save requirements.txt.
- Install the pinned dependencies by running this command in your terminal:
pip install -r requirements.txtThis may take a minute while pip downloads and installs all packages.
- Install one additional package for the traffic generator script you'll write later:
pip install requestsYou should see Successfully installed messages for both install commands.
Seeing install errors?
Make sure (venv) appears in your prompt before running pip. If you see permission errors, you likely forgot to activate the virtual environment.
If scikit-learn fails to install, confirm your Python version is 3.11 or higher. scikit-learn 1.9.0 requires Python 3.11+.
Need help? Help me fix my pip install errors in my virtual environment.
✔️ Awesome, I've got everything!
Great. Double check you've saved requirements.txt and both install commands completed without errors.
ⓧ I'd like to double check the full code
Here is the complete requirements.txt file:
fastapi==0.136.3
uvicorn==0.49.0
scikit-learn==1.9.0
prometheus-client==0.25.0
joblib==1.4.2
numpy==2.2.6
scipy==1.15.3
pydantic==2.11.4Verify your tools are ready
Your monitoring stack runs inside Docker Compose containers. Let's confirm both Docker and Python are available before you build anything.
- Check that Docker Compose is installed by running:
docker compose versionYou should see a version number printed, such as Docker Compose version v2.x.x.
- Confirm your Python version by running:
python3 --versionYou should see Python 3.11 or higher in the output.
Command not found?
If docker compose version fails, make sure Docker Desktop is running. The Docker CLI plugin requires the Docker daemon to be active.
If python3 --version shows a version below 3.11, you'll need to upgrade Python before continuing. scikit-learn 1.9.0 will not install on older versions.
Still stuck? Help me get Docker and Python 3.11+ working on my machine.
Your environment is ready. Next up, you'll train a model and build the production inference service that Prometheus will monitor.
Train the Model and Build the Inference Service
Your development environment is ready with all dependencies installed. Now you need a trained model and an API that serves predictions from it.
In this step, you will train a scikit-learn classifier, save it as a production artifact, and build a FastAPI inference service instrumented with 7 custom Prometheus metrics. The service will also detect data drift in real time by comparing incoming requests against training data.
In this step, get ready to:
- Train a RandomForest Iris classifier and save model artifacts for drift detection.
- Build a FastAPI inference service with 7 Prometheus metrics and 4 endpoints.
- Implement rolling-window drift detection using the Kolmogorov-Smirnov test.
Train the scikit-learn classifier and save model artifacts
Before the inference service can serve predictions, it needs a trained model to load. You will also save a copy of the training data so the service can compare future inputs against it for drift detection.
- Create a file called train_model.py in the root of your ml-monitoring project folder, then paste the following code:
import numpy as np
from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
import joblib
import json
import os
os.makedirs("model", exist_ok=True)
# Load and split data
iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(
iris.data, iris.target, test_size=0.2, random_state=42
)
# Train model
model = RandomForestClassifier(n_estimators=100, random_state=42)
model.fit(X_train, y_train)
# Save model artifact
joblib.dump(model, "model/iris_model.joblib")
# Save reference data for drift detection
np.save("model/reference_data.npy", X_train)
# Save model metadata
metadata = {
"model_version": "v1",
"algorithm": "RandomForestClassifier",
"n_estimators": 100,
"n_features": 4,
"feature_names": iris.feature_names.tolist(),
"target_names": iris.target_names.tolist(),
"training_samples": len(X_train),
"test_accuracy": float(model.score(X_test, y_test)),
}
with open("model/metadata.json", "w") as f:
json.dump(metadata, f, indent=2)
print(f"Model accuracy: {metadata['test_accuracy']:.4f}")
print(f"Training samples: {metadata['training_samples']}")
print("Artifacts saved:")
print(" model/iris_model.joblib")
print(" model/reference_data.npy")
print(" model/metadata.json")What does this script do?
This script loads the Iris dataset (4 features, 3 classes), splits it 80/20, and trains a RandomForestClassifier with 100 estimators.
It produces three artifacts. iris_model.joblib is the serialized model. reference_data.npy stores the 120 training samples so the service can compare future inputs for drift. metadata.json records version info, accuracy, and feature names the service uses at runtime.
- Run the training script by executing this command:
python train_model.py- Confirm the artifacts were created by running:
ls model/- You should see iris_model.joblib, metadata.json, and reference_data.npy in the output. The terminal also prints accuracy around 1.0000 and 120 training samples.
Seeing a ModuleNotFoundError?
Make sure your virtual environment is active. You should see (venv) at the start of your terminal prompt. If not, activate it again with source venv/bin/activate (macOS/Linux) or venv\Scripts\activate (Windows).
help me fix a ModuleNotFoundError when running train_model.py
Build the FastAPI service structure with Prometheus metrics
With the model artifacts saved, you can build the API that loads them and serves predictions. The service needs to expose Prometheus metrics so the monitoring stack can scrape performance data.
- Create an empty app/__init__.py file to make the app/ directory a Python package:
touch app/__init__.py- Create a file called app/main.py and paste the following imports and application setup:
import time
import uuid
import json
import numpy as np
import joblib
from scipy import stats
from fastapi import FastAPI, HTTPException, Request
from pydantic import BaseModel
from prometheus_client import (
Counter,
Histogram,
Gauge,
Info,
generate_latest,
CONTENT_TYPE_LATEST,
)
from fastapi.responses import Response, JSONResponse
app = FastAPI(title="ML Inference Service", version="1.0.0")
# Load model and reference data
model = joblib.load("model/iris_model.joblib")
reference_data = np.load("model/reference_data.npy")
with open("model/metadata.json") as f:
model_metadata = json.load(f)
MODEL_VERSION = model_metadata["model_version"]What does this code do?
The app loads the three model artifacts at startup. model is the trained classifier. reference_data holds the original training feature distributions for drift comparison. model_metadata provides version info and class names.
MODEL_VERSION is extracted from metadata so every metric is labelled with the model version. This becomes critical when you serve multiple model versions later.
- Below the code you just pasted, add the 7 Prometheus metric definitions:
# --- Prometheus Metrics ---
PREDICTION_COUNT = Counter(
"ml_predictions_total",
"Total number of predictions made",
["model_version", "predicted_class"],
)
PREDICTION_LATENCY = Histogram(
"ml_prediction_latency_seconds",
"Time spent processing prediction request",
["model_version"],
buckets=[0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0],
)
PREDICTION_ERRORS = Counter(
"ml_prediction_errors_total",
"Total number of prediction errors",
["model_version", "error_type"],
)
DRIFT_SCORE = Gauge(
"ml_data_drift_score",
"Data drift score (KS statistic) per feature",
["model_version", "feature"],
)
PREDICTION_CONFIDENCE = Gauge(
"ml_prediction_confidence",
"Most recent prediction confidence score",
["model_version"],
)
FEATURE_MEAN = Gauge(
"ml_feature_mean",
"Rolling mean of input features",
["model_version", "feature"],
)
MODEL_INFO = Info(
"ml_model",
"Model metadata information",
)
MODEL_INFO.info({
"version": MODEL_VERSION,
"algorithm": model_metadata["algorithm"],
"training_samples": str(model_metadata["training_samples"]),
"test_accuracy": str(model_metadata["test_accuracy"]),
})What are these metric types?
A Counter only goes up. It tracks totals like prediction count and error count. A Histogram records the distribution of values (latency) in configurable buckets. A Gauge can go up or down, perfect for current state like drift score and confidence.
The Info metric exposes static metadata as key-value pairs. Labels like model_version and predicted_class let you filter and aggregate in PromQL queries later.
- Continue in app/main.py by adding the drift detection state and Pydantic request/response models:
# --- Drift Detection State ---
WINDOW_SIZE = 50
recent_features: list[list[float]] = []
# --- Request/Response Models ---
class PredictionRequest(BaseModel):
features: list[float]
class PredictionResponse(BaseModel):
prediction: int
prediction_label: str
confidence: float
model_version: str
request_id: strWhy a rolling window?
WINDOW_SIZE controls how many recent predictions are stored for drift comparison. After 50 requests accumulate, the service compares the feature distributions of recent inputs against the training data. This detects gradual changes in input patterns without needing to store every request forever.
- Add the global exception handler and utility endpoints below the models:
# --- Exception Handler ---
@app.exception_handler(Exception)
async def global_exception_handler(request: Request, exc: Exception):
PREDICTION_ERRORS.labels(
model_version=MODEL_VERSION, error_type="unhandled"
).inc()
return JSONResponse(
status_code=500,
content={"detail": "Internal server error", "type": "unhandled"},
)
# --- Endpoints ---
@app.get("/health")
def health():
return {
"status": "healthy",
"model_version": MODEL_VERSION,
"model_algorithm": model_metadata["algorithm"],
}
@app.get("/ready")
def ready():
try:
test_input = np.array([[5.1, 3.5, 1.4, 0.2]])
model.predict(test_input)
return {"status": "ready", "model_loaded": True}
except Exception:
raise HTTPException(status_code=503, detail="Model not ready")What do these endpoints do?
The exception handler catches any unhandled error and increments the PREDICTION_ERRORS counter before returning a 500. This ensures every failure is captured in metrics.
The /health endpoint confirms the service is running. The /ready endpoint actually runs a test prediction to verify the model can produce output. Docker health checks and Prometheus will poll these endpoints.
- Save app/main.py so far. You will continue adding to it in the next substep.
Implement drift detection in the prediction endpoint
The prediction endpoint is where the core business logic lives. It validates input, runs the model, records all metrics, and calculates drift using the Kolmogorov-Smirnov test after each request.
- At the bottom of app/main.py, add the /predict endpoint and the /metrics endpoint:
@app.post("/predict", response_model=PredictionResponse)
def predict(request: PredictionRequest):
request_id = str(uuid.uuid4())
start_time = time.time()
# Validate input
if len(request.features) != 4:
PREDICTION_ERRORS.labels(
model_version=MODEL_VERSION, error_type="validation"
).inc()
raise HTTPException(
status_code=422,
detail=f"Expected 4 features, got {len(request.features)}",
)
features = np.array(request.features).reshape(1, -1)
# Check for NaN or Inf
if not np.isfinite(features).all():
PREDICTION_ERRORS.labels(
model_version=MODEL_VERSION, error_type="invalid_values"
).inc()
raise HTTPException(
status_code=422, detail="Features contain NaN or Inf values"
)
# Make prediction
prediction = int(model.predict(features)[0])
probabilities = model.predict_proba(features)[0]
confidence = float(probabilities.max())
prediction_label = model_metadata["target_names"][prediction]
# Record latency
latency = time.time() - start_time
PREDICTION_LATENCY.labels(model_version=MODEL_VERSION).observe(latency)
# Record prediction count
PREDICTION_COUNT.labels(
model_version=MODEL_VERSION, predicted_class=str(prediction)
).inc()
# Record confidence
PREDICTION_CONFIDENCE.labels(model_version=MODEL_VERSION).set(confidence)
# Update drift detection window
recent_features.append(request.features)
if len(recent_features) > WINDOW_SIZE:
recent_features.pop(0)
# Calculate drift and feature statistics
if len(recent_features) >= WINDOW_SIZE:
recent_array = np.array(recent_features)
for i in range(4):
# KS drift score
ks_stat, _ = stats.ks_2samp(reference_data[:, i], recent_array[:, i])
feature_name = f"feature_{i}"
DRIFT_SCORE.labels(
model_version=MODEL_VERSION, feature=feature_name
).set(ks_stat)
# Feature mean tracking
FEATURE_MEAN.labels(
model_version=MODEL_VERSION, feature=feature_name
).set(float(recent_array[:, i].mean()))
return PredictionResponse(
prediction=prediction,
prediction_label=prediction_label,
confidence=confidence,
model_version=MODEL_VERSION,
request_id=request_id,
)
@app.get("/metrics")
def metrics():
return Response(content=generate_latest(), media_type=CONTENT_TYPE_LATEST)How does the KS drift detection work?
After the rolling window fills to 50 samples, the code calls stats.ks_2samp for each of the 4 features. This compares the distribution of recent inputs against the original training data. The KS statistic ranges from 0 (identical distributions) to 1 (completely different).
When inputs shift away from the training distribution, the KS score rises. Prometheus will alert when any feature crosses the 0.3 threshold. The /metrics endpoint exposes all 7 metrics in Prometheus text format for scraping.
- Save app/main.py.
- Start the service locally to verify everything loads correctly by running:
uvicorn app.main:app --host 0.0.0.0 --port 8000- You should see output indicating Uvicorn is running on http://0.0.0.0:8000.
- Open a second terminal (keep the server running), activate your virtual environment, and test the health endpoint by running:
curl http://localhost:8000/health- You should see a JSON response with "status": "healthy" and "model_version": "v1".
- Stop the server with Ctrl+C in the first terminal.
Seeing an ImportError or the server won't start?
Double check that app/__init__.py exists. Without it, Python cannot import app.main as a package. Also verify you are running the command from the project root directory (the folder containing both app/ and model/).
help me fix an error when starting my FastAPI ML inference service with uvicorn
✔️ Awesome, I've got everything!
Great. Double check you have saved both train_model.py and app/main.py.
ⓧ I'd like to double check the full code
Here is the complete app/main.py for comparison:
import time
import uuid
import json
import numpy as np
import joblib
from scipy import stats
from fastapi import FastAPI, HTTPException, Request
from pydantic import BaseModel
from prometheus_client import (
Counter,
Histogram,
Gauge,
Info,
generate_latest,
CONTENT_TYPE_LATEST,
)
from fastapi.responses import Response, JSONResponse
app = FastAPI(title="ML Inference Service", version="1.0.0")
# Load model and reference data
model = joblib.load("model/iris_model.joblib")
reference_data = np.load("model/reference_data.npy")
with open("model/metadata.json") as f:
model_metadata = json.load(f)
MODEL_VERSION = model_metadata["model_version"]
# --- Prometheus Metrics ---
PREDICTION_COUNT = Counter(
"ml_predictions_total",
"Total number of predictions made",
["model_version", "predicted_class"],
)
PREDICTION_LATENCY = Histogram(
"ml_prediction_latency_seconds",
"Time spent processing prediction request",
["model_version"],
buckets=[0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0],
)
PREDICTION_ERRORS = Counter(
"ml_prediction_errors_total",
"Total number of prediction errors",
["model_version", "error_type"],
)
DRIFT_SCORE = Gauge(
"ml_data_drift_score",
"Data drift score (KS statistic) per feature",
["model_version", "feature"],
)
PREDICTION_CONFIDENCE = Gauge(
"ml_prediction_confidence",
"Most recent prediction confidence score",
["model_version"],
)
FEATURE_MEAN = Gauge(
"ml_feature_mean",
"Rolling mean of input features",
["model_version", "feature"],
)
MODEL_INFO = Info(
"ml_model",
"Model metadata information",
)
MODEL_INFO.info({
"version": MODEL_VERSION,
"algorithm": model_metadata["algorithm"],
"training_samples": str(model_metadata["training_samples"]),
"test_accuracy": str(model_metadata["test_accuracy"]),
})
# --- Drift Detection State ---
WINDOW_SIZE = 50
recent_features: list[list[float]] = []
# --- Request/Response Models ---
class PredictionRequest(BaseModel):
features: list[float]
class PredictionResponse(BaseModel):
prediction: int
prediction_label: str
confidence: float
model_version: str
request_id: str
# --- Exception Handler ---
@app.exception_handler(Exception)
async def global_exception_handler(request: Request, exc: Exception):
PREDICTION_ERRORS.labels(
model_version=MODEL_VERSION, error_type="unhandled"
).inc()
return JSONResponse(
status_code=500,
content={"detail": "Internal server error", "type": "unhandled"},
)
# --- Endpoints ---
@app.get("/health")
def health():
return {
"status": "healthy",
"model_version": MODEL_VERSION,
"model_algorithm": model_metadata["algorithm"],
}
@app.get("/ready")
def ready():
try:
test_input = np.array([[5.1, 3.5, 1.4, 0.2]])
model.predict(test_input)
return {"status": "ready", "model_loaded": True}
except Exception:
raise HTTPException(status_code=503, detail="Model not ready")
@app.post("/predict", response_model=PredictionResponse)
def predict(request: PredictionRequest):
request_id = str(uuid.uuid4())
start_time = time.time()
# Validate input
if len(request.features) != 4:
PREDICTION_ERRORS.labels(
model_version=MODEL_VERSION, error_type="validation"
).inc()
raise HTTPException(
status_code=422,
detail=f"Expected 4 features, got {len(request.features)}",
)
features = np.array(request.features).reshape(1, -1)
# Check for NaN or Inf
if not np.isfinite(features).all():
PREDICTION_ERRORS.labels(
model_version=MODEL_VERSION, error_type="invalid_values"
).inc()
raise HTTPException(
status_code=422, detail="Features contain NaN or Inf values"
)
# Make prediction
prediction = int(model.predict(features)[0])
probabilities = model.predict_proba(features)[0]
confidence = float(probabilities.max())
prediction_label = model_metadata["target_names"][prediction]
# Record latency
latency = time.time() - start_time
PREDICTION_LATENCY.labels(model_version=MODEL_VERSION).observe(latency)
# Record prediction count
PREDICTION_COUNT.labels(
model_version=MODEL_VERSION, predicted_class=str(prediction)
).inc()
# Record confidence
PREDICTION_CONFIDENCE.labels(model_version=MODEL_VERSION).set(confidence)
# Update drift detection window
recent_features.append(request.features)
if len(recent_features) > WINDOW_SIZE:
recent_features.pop(0)
# Calculate drift and feature statistics
if len(recent_features) >= WINDOW_SIZE:
recent_array = np.array(recent_features)
for i in range(4):
# KS drift score
ks_stat, _ = stats.ks_2samp(reference_data[:, i], recent_array[:, i])
feature_name = f"feature_{i}"
DRIFT_SCORE.labels(
model_version=MODEL_VERSION, feature=feature_name
).set(ks_stat)
# Feature mean tracking
FEATURE_MEAN.labels(
model_version=MODEL_VERSION, feature=feature_name
).set(float(recent_array[:, i].mean()))
return PredictionResponse(
prediction=prediction,
prediction_label=prediction_label,
confidence=confidence,
model_version=MODEL_VERSION,
request_id=request_id,
)
@app.get("/metrics")
def metrics():
return Response(content=generate_latest(), media_type=CONTENT_TYPE_LATEST)Your ML inference service is live and instrumented. Next up, you will deploy the full observability stack with Docker Compose so Prometheus can scrape these metrics and Alertmanager can fire alerts when drift occurs.
Deploy the Observability Stack with Docker Compose
Your ML inference service is instrumented and ready to expose metrics. Now you need to package it into a container and deploy it alongside the monitoring tools that will collect, store, alert on, and visualize those metrics.
In this step, you will containerize the inference service with a production-optimized Dockerfile, then orchestrate four services with Docker Compose: the model server, Prometheus for metrics collection and alerting rules, Grafana for dashboards, and Alertmanager for alert routing.
In this step, get ready to:
- Write a multi-stage Dockerfile and Docker Compose file orchestrating 4 services.
- Configure Prometheus with scrape targets, recording rules, and alert rules.
- Configure Alertmanager with severity-based routing and inhibition rules.
Write the multi-stage Dockerfile and Docker Compose configuration
A multi-stage Dockerfile keeps the final image small by separating the build step (installing packages) from the runtime step (running the app). This matters for ML services where dependency installation pulls in large compilation artifacts you do not need at runtime.
- Create the Dockerfile in the root of your ml-monitoring project folder by pasting the following code:
FROM python:3.11-slim AS builder
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir --prefix=/install -r requirements.txt
FROM python:3.11-slim
WORKDIR /app
# Copy installed packages from builder stage
COPY --from=builder /install /usr/local
# Copy application code and model artifacts
COPY model/ model/
COPY app/ app/
# Run as non-root user for security
RUN useradd --create-home appuser
USER appuser
EXPOSE 8000
# Health check so Docker knows when the container is ready
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')"
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000", "--workers", "1"]What does this Dockerfile do?
- The builder stage installs Python packages into a clean prefix, then the final stage copies only the installed packages. This avoids shipping pip cache and build tools.
- The appuser ensures the container runs as a non-root user, which is a production security best practice.
- The HEALTHCHECK instruction lets Docker and Docker Compose know when the service is actually ready to serve requests.
- Save Dockerfile.
Seeing a syntax highlight issue?
Make sure the file is named exactly Dockerfile with a capital D and no file extension. Some editors may try to add .txt automatically.
Help me debug my Dockerfile
Now you need a Docker Compose file to orchestrate the model server alongside Prometheus, Grafana, and Alertmanager.
- Create docker-compose.yml in the root of your project folder by pasting the following code:
services:
model-server:
build: .
ports:
- "8000:8000"
restart: unless-stopped
healthcheck:
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')"]
interval: 15s
timeout: 5s
retries: 3
start_period: 15s
prometheus:
image: prom/prometheus:v3.12.0
ports:
- "9090:9090"
restart: unless-stopped
volumes:
- ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
- ./prometheus/rules.yml:/etc/prometheus/rules.yml
- prometheus-data:/prometheus
command:
- "--config.file=/etc/prometheus/prometheus.yml"
- "--storage.tsdb.retention.time=7d"
- "--web.enable-lifecycle"
depends_on:
model-server:
condition: service_healthy
grafana:
image: grafana/grafana:13.0.2
ports:
- "3000:3000"
restart: unless-stopped
environment:
- GF_AUTH_ANONYMOUS_ENABLED=true
- GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
- GF_AUTH_BASIC_ENABLED=false
- GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH=/var/lib/grafana/dashboards/ml-monitoring.json
volumes:
- ./grafana/provisioning:/etc/grafana/provisioning
- ./grafana/dashboards:/var/lib/grafana/dashboards
- grafana-data:/var/lib/grafana
depends_on:
prometheus:
condition: service_started
alertmanager:
image: prom/alertmanager:v0.32.2
ports:
- "9093:9093"
restart: unless-stopped
volumes:
- ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml
command:
- "--config.file=/etc/alertmanager/alertmanager.yml"
- "--storage.path=/alertmanager"
- "--log.level=info"
volumes:
prometheus-data:
grafana-data:What does this compose file do?
- The model-server builds from your Dockerfile and exposes port 8000. Its healthcheck lets other services wait until it is ready.
- The prometheus service uses depends_on: service_healthy so it only starts scraping after the model server passes its health check.
- The grafana service uses anonymous auth with Admin role so you can access dashboards without a login prompt during local development.
- The alertmanager service receives firing alerts from Prometheus and routes them based on severity labels.
- Named volumes (prometheus-data, grafana-data) persist metrics and dashboard state across container restarts.
- Save docker-compose.yml.
- Validate the compose file syntax by running:
docker compose config --quietIf the file is valid, you see no output. If there is a YAML syntax error, you see an error message pointing to the problem line.
Seeing a YAML parsing error?
YAML is indentation-sensitive. Make sure you are using spaces (not tabs) and that nested keys are indented exactly 2 spaces per level.
Help me fix my docker-compose.yml
✔️ Awesome, I've got everything!
Great. Make sure both Dockerfile and docker-compose.yml are saved in the root of your project folder.
ⓧ I'd like to double check the full code
Here is the complete Dockerfile:
FROM python:3.11-slim AS builder
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir --prefix=/install -r requirements.txt
FROM python:3.11-slim
WORKDIR /app
COPY --from=builder /install /usr/local
COPY model/ model/
COPY app/ app/
RUN useradd --create-home appuser
USER appuser
EXPOSE 8000
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')"
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000", "--workers", "1"]Here is the complete docker-compose.yml:
services:
model-server:
build: .
ports:
- "8000:8000"
restart: unless-stopped
healthcheck:
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')"]
interval: 15s
timeout: 5s
retries: 3
start_period: 15s
prometheus:
image: prom/prometheus:v3.12.0
ports:
- "9090:9090"
restart: unless-stopped
volumes:
- ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
- ./prometheus/rules.yml:/etc/prometheus/rules.yml
- prometheus-data:/prometheus
command:
- "--config.file=/etc/prometheus/prometheus.yml"
- "--storage.tsdb.retention.time=7d"
- "--web.enable-lifecycle"
depends_on:
model-server:
condition: service_healthy
grafana:
image: grafana/grafana:13.0.2
ports:
- "3000:3000"
restart: unless-stopped
environment:
- GF_AUTH_ANONYMOUS_ENABLED=true
- GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
- GF_AUTH_BASIC_ENABLED=false
- GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH=/var/lib/grafana/dashboards/ml-monitoring.json
volumes:
- ./grafana/provisioning:/etc/grafana/provisioning
- ./grafana/dashboards:/var/lib/grafana/dashboards
- grafana-data:/var/lib/grafana
depends_on:
prometheus:
condition: service_started
alertmanager:
image: prom/alertmanager:v0.32.2
ports:
- "9093:9093"
restart: unless-stopped
volumes:
- ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml
command:
- "--config.file=/etc/alertmanager/alertmanager.yml"
- "--storage.path=/alertmanager"
- "--log.level=info"
volumes:
prometheus-data:
grafana-data:Configure Prometheus scraping and alert rules
Prometheus needs two configuration files: one that tells it where to scrape metrics from, and one that defines recording rules (pre-computed queries) and alert rules (conditions that fire alerts).
- Create the prometheus directory by running:
mkdir -p prometheus- Create prometheus/prometheus.yml and paste the following configuration:
global:
scrape_interval: 15s
evaluation_interval: 15s
rule_files:
- "rules.yml"
alerting:
alertmanagers:
- static_configs:
- targets:
- "alertmanager:9093"
scrape_configs:
- job_name: "ml-model-server"
metrics_path: "/metrics"
scrape_interval: 10s
static_configs:
- targets: ["model-server:8000"]
labels:
environment: "development"
service: "ml-inference"What does this configuration do?
- The scrape_interval: 15s sets how often Prometheus collects metrics globally. The evaluation_interval: 15s controls how often recording rules and alert rules are evaluated.
- The rule_files section points to rules.yml which you will create next.
- The alerting section tells Prometheus to send firing alerts to Alertmanager at alertmanager:9093 (the Docker Compose service name).
- The scrape job targets model-server:8000 with a 10-second interval for higher-resolution model metrics.
- Save prometheus/prometheus.yml.
Now create the rules file. Recording rules pre-compute expensive PromQL queries so dashboards load instantly. Alert rules define conditions that trigger notifications.
- Create prometheus/rules.yml and paste the following code:
groups:
- name: ml_recording_rules
interval: 15s
rules:
- record: ml:prediction_latency_p95:5m
expr: histogram_quantile(0.95, sum by (le, model_version) (rate(ml_prediction_latency_seconds_bucket[5m])))
- record: ml:prediction_latency_p50:5m
expr: histogram_quantile(0.50, sum by (le, model_version) (rate(ml_prediction_latency_seconds_bucket[5m])))
- record: ml:prediction_rate:5m
expr: sum by (model_version) (rate(ml_predictions_total[5m]))
- record: ml:error_rate:5m
expr: sum by (model_version) (rate(ml_prediction_errors_total[5m])) / (sum by (model_version) (rate(ml_predictions_total[5m])) + sum by (model_version) (rate(ml_prediction_errors_total[5m])))
- name: ml_model_performance_alerts
interval: 15s
rules:
- alert: HighPredictionLatency
expr: ml:prediction_latency_p95:5m > 0.5
for: 2m
labels:
severity: warning
team: ml-platform
component: inference
annotations:
summary: "High prediction latency detected"
description: "P95 latency for model {{ $labels.model_version }} is {{ $value | printf \"%.3f\" }}s, exceeding 500ms threshold."
runbook: "Check model-server resource usage. Consider scaling or optimizing model."
- alert: LowPredictionConfidence
expr: ml_prediction_confidence < 0.6
for: 3m
labels:
severity: warning
team: ml-platform
component: model-quality
annotations:
summary: "Low prediction confidence"
description: "Model {{ $labels.model_version }} confidence dropped to {{ $value | printf \"%.3f\" }}."
runbook: "Investigate input data quality. May indicate out-of-distribution inputs."
- name: ml_data_quality_alerts
interval: 15s
rules:
- alert: DataDriftDetected
expr: max by (model_version) (ml_data_drift_score) > 0.3
for: 1m
labels:
severity: critical
team: ml-platform
component: data-quality
annotations:
summary: "Data drift detected in model inputs"
description: "KS drift score for model {{ $labels.model_version }} is {{ $value | printf \"%.3f\" }}, exceeding threshold 0.3."
runbook: "Investigate input data source. Consider retraining model with recent data."
- alert: FeatureDistributionShift
expr: count by (model_version) (ml_data_drift_score > 0.2) >= 3
for: 2m
labels:
severity: warning
team: ml-platform
component: data-quality
annotations:
summary: "Multiple features showing distribution shift"
description: "Model {{ $labels.model_version }} has {{ $value }} features with drift score above 0.2."
runbook: "Multiple features drifting indicates systematic input change. Review data pipeline."
- name: ml_service_health_alerts
interval: 15s
rules:
- alert: MLServiceDown
expr: up{job="ml-model-server"} == 0
for: 30s
labels:
severity: critical
team: ml-platform
component: infrastructure
annotations:
summary: "ML inference service is down"
description: "The ml-model-server target has been unreachable for 30 seconds."
runbook: "Check container health: docker compose logs model-server. Restart if needed."
- alert: HighErrorRate
expr: ml:error_rate:5m > 0.05
for: 2m
labels:
severity: critical
team: ml-platform
component: inference
annotations:
summary: "High prediction error rate"
description: "Model {{ $labels.model_version }} error rate is {{ $value | printf \"%.1f\" }}%%, exceeding 5%% threshold."
runbook: "Check error types in logs. May indicate bad input data or model corruption."What do these rules do?
The file has two types of rules organized into 4 groups.
Recording rules (ml_recording_rules) pre-compute p95 latency, p50 latency, prediction rate, and error rate every 15 seconds. Grafana queries these pre-computed values instead of recalculating expensive histogram quantiles on every dashboard refresh.
Alert rules are split into 3 groups by domain: Model Performance (HighPredictionLatency, LowPredictionConfidence), Data Quality (DataDriftDetected, FeatureDistributionShift), and Service Health (MLServiceDown, HighErrorRate). Each alert carries a severity label that Alertmanager uses for routing.
- Save prometheus/rules.yml.
- Confirm both files exist by running:
ls prometheus/You should see prometheus.yml and rules.yml listed.
Missing one of the files?
Make sure you created both files inside the prometheus/ directory, not in the project root. If you see them in the wrong place, move them with mv prometheus.yml prometheus/.
Help me fix my prometheus directory structure
✔️ Awesome, I've got everything!
Great. Make sure both files are saved inside the prometheus/ directory.
ⓧ I'd like to double check the full code
Here is the complete prometheus/prometheus.yml:
global:
scrape_interval: 15s
evaluation_interval: 15s
rule_files:
- "rules.yml"
alerting:
alertmanagers:
- static_configs:
- targets:
- "alertmanager:9093"
scrape_configs:
- job_name: "ml-model-server"
metrics_path: "/metrics"
scrape_interval: 10s
static_configs:
- targets: ["model-server:8000"]
labels:
environment: "development"
service: "ml-inference"Here is the complete prometheus/rules.yml:
groups:
- name: ml_recording_rules
interval: 15s
rules:
- record: ml:prediction_latency_p95:5m
expr: histogram_quantile(0.95, sum by (le, model_version) (rate(ml_prediction_latency_seconds_bucket[5m])))
- record: ml:prediction_latency_p50:5m
expr: histogram_quantile(0.50, sum by (le, model_version) (rate(ml_prediction_latency_seconds_bucket[5m])))
- record: ml:prediction_rate:5m
expr: sum by (model_version) (rate(ml_predictions_total[5m]))
- record: ml:error_rate:5m
expr: sum by (model_version) (rate(ml_prediction_errors_total[5m])) / (sum by (model_version) (rate(ml_predictions_total[5m])) + sum by (model_version) (rate(ml_prediction_errors_total[5m])))
- name: ml_model_performance_alerts
interval: 15s
rules:
- alert: HighPredictionLatency
expr: ml:prediction_latency_p95:5m > 0.5
for: 2m
labels:
severity: warning
team: ml-platform
component: inference
annotations:
summary: "High prediction latency detected"
description: "P95 latency for model {{ $labels.model_version }} is {{ $value | printf \"%.3f\" }}s, exceeding 500ms threshold."
runbook: "Check model-server resource usage. Consider scaling or optimizing model."
- alert: LowPredictionConfidence
expr: ml_prediction_confidence < 0.6
for: 3m
labels:
severity: warning
team: ml-platform
component: model-quality
annotations:
summary: "Low prediction confidence"
description: "Model {{ $labels.model_version }} confidence dropped to {{ $value | printf \"%.3f\" }}."
runbook: "Investigate input data quality. May indicate out-of-distribution inputs."
- name: ml_data_quality_alerts
interval: 15s
rules:
- alert: DataDriftDetected
expr: max by (model_version) (ml_data_drift_score) > 0.3
for: 1m
labels:
severity: critical
team: ml-platform
component: data-quality
annotations:
summary: "Data drift detected in model inputs"
description: "KS drift score for model {{ $labels.model_version }} is {{ $value | printf \"%.3f\" }}, exceeding threshold 0.3."
runbook: "Investigate input data source. Consider retraining model with recent data."
- alert: FeatureDistributionShift
expr: count by (model_version) (ml_data_drift_score > 0.2) >= 3
for: 2m
labels:
severity: warning
team: ml-platform
component: data-quality
annotations:
summary: "Multiple features showing distribution shift"
description: "Model {{ $labels.model_version }} has {{ $value }} features with drift score above 0.2."
runbook: "Multiple features drifting indicates systematic input change. Review data pipeline."
- name: ml_service_health_alerts
interval: 15s
rules:
- alert: MLServiceDown
expr: up{job="ml-model-server"} == 0
for: 30s
labels:
severity: critical
team: ml-platform
component: infrastructure
annotations:
summary: "ML inference service is down"
description: "The ml-model-server target has been unreachable for 30 seconds."
runbook: "Check container health: docker compose logs model-server. Restart if needed."
- alert: HighErrorRate
expr: ml:error_rate:5m > 0.05
for: 2m
labels:
severity: critical
team: ml-platform
component: inference
annotations:
summary: "High prediction error rate"
description: "Model {{ $labels.model_version }} error rate is {{ $value | printf \"%.1f\" }}%%, exceeding 5%% threshold."
runbook: "Check error types in logs. May indicate bad input data or model corruption."Configure Alertmanager routing and inhibition rules
Alertmanager receives firing alerts from Prometheus and decides where to send them. A routing tree matches alert labels (like severity) to receivers. Inhibition rules prevent redundant alerts from firing when a higher-severity alert already covers the issue.
- Create the alertmanager directory by running:
mkdir -p alertmanager- Create alertmanager/alertmanager.yml and paste the following configuration:
global:
resolve_timeout: 5m
route:
group_by: ["alertname", "model_version"]
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
receiver: "webhook-info"
routes:
- matchers:
- severity = critical
receiver: "webhook-critical"
group_wait: 10s
repeat_interval: 1h
continue: false
- matchers:
- severity = warning
receiver: "webhook-warning"
group_wait: 30s
repeat_interval: 2h
continue: false
inhibit_rules:
- source_matchers:
- severity = critical
- component = data-quality
target_matchers:
- severity = warning
- component = model-quality
equal: ["model_version"]
- source_matchers:
- alertname = MLServiceDown
target_matchers:
- severity = warning
equal: []
receivers:
- name: "webhook-critical"
webhook_configs:
- url: "http://model-server:8000/health"
send_resolved: true
- name: "webhook-warning"
webhook_configs:
- url: "http://model-server:8000/health"
send_resolved: true
- name: "webhook-info"
webhook_configs:
- url: "http://model-server:8000/health"
send_resolved: trueWhat does this configuration do?
The route section groups alerts by alertname and model_version, then matches severity labels to route critical alerts to webhook-critical (with a faster 10s group wait) and warnings to webhook-warning.
The first inhibition rule says: when a critical data-quality alert is firing, suppress warning model-quality alerts for the same model_version. This prevents a LowPredictionConfidence warning when DataDriftDetected already explains the problem.
The second inhibition rule says: when MLServiceDown is firing, suppress all warning-level alerts. If the service is unreachable, other metrics are unreliable.
Why webhook receivers pointing to the health endpoint?
In production, receivers would point to Slack, PagerDuty, or a custom incident management webhook. For this local setup, the receivers point to the model server's health endpoint as a harmless target that accepts HTTP requests.
You can replace the URL with a service like webhook.site to inspect the actual alert payloads Alertmanager sends.
- Save alertmanager/alertmanager.yml.
- Confirm the file exists by running:
ls alertmanager/You should see alertmanager.yml listed.
File not showing up?
Double-check that you created the file inside the alertmanager/ directory. The Docker Compose volume mount expects the file at exactly ./alertmanager/alertmanager.yml.
Help me fix my alertmanager file location
✔️ Awesome, I've got everything!
Great. Make sure alertmanager/alertmanager.yml is saved.
ⓧ I'd like to double check the full code
Here is the complete alertmanager/alertmanager.yml:
global:
resolve_timeout: 5m
route:
group_by: ["alertname", "model_version"]
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
receiver: "webhook-info"
routes:
- matchers:
- severity = critical
receiver: "webhook-critical"
group_wait: 10s
repeat_interval: 1h
continue: false
- matchers:
- severity = warning
receiver: "webhook-warning"
group_wait: 30s
repeat_interval: 2h
continue: false
inhibit_rules:
- source_matchers:
- severity = critical
- component = data-quality
target_matchers:
- severity = warning
- component = model-quality
equal: ["model_version"]
- source_matchers:
- alertname = MLServiceDown
target_matchers:
- severity = warning
equal: []
receivers:
- name: "webhook-critical"
webhook_configs:
- url: "http://model-server:8000/health"
send_resolved: true
- name: "webhook-warning"
webhook_configs:
- url: "http://model-server:8000/health"
send_resolved: true
- name: "webhook-info"
webhook_configs:
- url: "http://model-server:8000/health"
send_resolved: trueYour entire observability stack is now defined as code. Next up, you will provision a Grafana dashboard and launch all four services to see them working together.
Provision the Grafana Dashboard as Code
You have a fully configured alerting pipeline with Prometheus rules and Alertmanager routing. But alerts alone are not enough.
Production teams need visual dashboards to spot trends before alerts fire. In this step, you will provision a Grafana dashboard entirely as code. No manual clicking through the UI. Every panel, datasource, and threshold lives in version-controlled files that deploy automatically when the stack starts.
In this step, get ready to:
- Create Grafana provisioning files for the Prometheus datasource.
- Build an 8-panel monitoring dashboard as a JSON file.
- Launch the full Docker Compose stack and verify all 4 services are healthy.
Create the Grafana datasource provisioning file
Grafana needs to know where to find Prometheus. Instead of configuring this manually in the Grafana UI, you define it in a YAML file that Grafana reads on startup.
- Create the Grafana provisioning directory structure by running these commands:
mkdir -p grafana/provisioning/datasources
mkdir -p grafana/provisioning/dashboards
mkdir -p grafana/dashboards- Confirm the directories exist by running:
ls grafana/provisioningYou should see dashboards and datasources listed in the output.
- Create a new file at grafana/provisioning/datasources/datasource.yml and add the following code:
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
access: proxy
url: http://prometheus:9090
isDefault: true
editable: false
jsonData:
timeInterval: "10s"What does this configuration do?
- name: Prometheus registers the datasource with a human-readable name in the Grafana UI.
- access: proxy means Grafana proxies queries to Prometheus on behalf of the browser. The browser never talks to Prometheus directly.
- url: http://prometheus:9090 uses the Docker Compose service name as the hostname. Containers on the same network resolve each other by service name.
- timeInterval: "10s" tells Grafana the minimum scrape interval so it can auto-adjust query step sizes.
- Save grafana/provisioning/datasources/datasource.yml.
File not saving in the right place?
Make sure the file path is exactly grafana/provisioning/datasources/datasource.yml relative to your project root. If you accidentally put it in grafana/datasources/ (missing provisioning/), Grafana will not find it.
help me debug my Grafana datasource provisioning file location
Create the dashboard provisioning and JSON definition
Grafana also needs a provider configuration that tells it where to find dashboard JSON files on disk. Then you create the actual dashboard definition with 8 monitoring panels covering every aspect of your ML service.
- Create a new file at grafana/provisioning/dashboards/dashboards.yml and add this provisioning config:
apiVersion: 1
providers:
- name: "ML Monitoring"
orgId: 1
folder: "ML Monitoring"
type: file
disableDeletion: false
updateIntervalSeconds: 10
allowUiUpdates: true
options:
path: /var/lib/grafana/dashboards
foldersFromFilesStructure: falseWhat does this provider do?
This tells Grafana to look for JSON dashboard files at /var/lib/grafana/dashboards inside the container. Your docker-compose.yml already mounts ./grafana/dashboards to that path.
updateIntervalSeconds: 10 means Grafana re-reads the file every 10 seconds. Any change you make to the JSON on your host machine appears in Grafana automatically.
- Save grafana/provisioning/dashboards/dashboards.yml.
Now for the main event: an 8-panel dashboard that visualizes request rate, latency, predictions, drift, confidence, errors, feature statistics, and active alerts.
- Create a new file at grafana/dashboards/ml-monitoring.json and add this dashboard definition:
{
"annotations": {"list": []},
"editable": true,
"id": null,
"title": "ML Model Monitoring",
"uid": "ml-monitoring-v1",
"panels": [
{
"id": 1,
"title": "Prediction Request Rate (req/s)",
"type": "timeseries",
"gridPos": {"h": 7, "w": 12, "x": 0, "y": 0},
"datasource": {"type": "prometheus", "uid": ""},
"targets": [
{
"expr": "sum(rate(ml_predictions_total[5m])) by (model_version)",
"legendFormat": "{{model_version}}"
}
],
"fieldConfig": {"defaults": {"unit": "reqps"}}
},
{
"id": 2,
"title": "Prediction Latency (p95 / p50)",
"type": "timeseries",
"gridPos": {"h": 7, "w": 12, "x": 12, "y": 0},
"datasource": {"type": "prometheus", "uid": ""},
"targets": [
{
"expr": "ml:prediction_latency_p95:5m",
"legendFormat": "p95 {{model_version}}"
},
{
"expr": "ml:prediction_latency_p50:5m",
"legendFormat": "p50 {{model_version}}"
}
],
"fieldConfig": {"defaults": {"unit": "s"}}
},
{
"id": 3,
"title": "Prediction Class Distribution",
"type": "timeseries",
"gridPos": {"h": 7, "w": 12, "x": 0, "y": 7},
"datasource": {"type": "prometheus", "uid": ""},
"targets": [
{
"expr": "sum(rate(ml_predictions_total[5m])) by (predicted_class)",
"legendFormat": "class {{predicted_class}}"
}
]
},
{
"id": 4,
"title": "Data Drift Score (KS Statistic)",
"type": "timeseries",
"gridPos": {"h": 7, "w": 12, "x": 12, "y": 7},
"datasource": {"type": "prometheus", "uid": ""},
"targets": [
{
"expr": "ml_data_drift_score",
"legendFormat": "{{feature}} ({{model_version}})"
}
],
"fieldConfig": {
"defaults": {
"max": 1,
"min": 0,
"thresholds": {
"mode": "absolute",
"steps": [
{"color": "green", "value": null},
{"color": "yellow", "value": 0.2},
{"color": "red", "value": 0.3}
]
},
"custom": {"thresholdsStyle": {"mode": "line"}}
}
}
},
{
"id": 5,
"title": "Prediction Confidence",
"type": "gauge",
"gridPos": {"h": 7, "w": 6, "x": 0, "y": 14},
"datasource": {"type": "prometheus", "uid": ""},
"targets": [
{
"expr": "ml_prediction_confidence",
"legendFormat": "{{model_version}}"
}
],
"fieldConfig": {
"defaults": {
"min": 0,
"max": 1,
"thresholds": {
"mode": "absolute",
"steps": [
{"color": "red", "value": null},
{"color": "yellow", "value": 0.6},
{"color": "green", "value": 0.8}
]
}
}
}
},
{
"id": 6,
"title": "Error Rate (%)",
"type": "stat",
"gridPos": {"h": 7, "w": 6, "x": 6, "y": 14},
"datasource": {"type": "prometheus", "uid": ""},
"targets": [
{
"expr": "ml:error_rate:5m * 100",
"legendFormat": "{{model_version}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "percent",
"thresholds": {
"mode": "absolute",
"steps": [
{"color": "green", "value": null},
{"color": "yellow", "value": 2},
{"color": "red", "value": 5}
]
}
}
}
},
{
"id": 7,
"title": "Feature Mean (Rolling Window)",
"type": "timeseries",
"gridPos": {"h": 7, "w": 12, "x": 12, "y": 14},
"datasource": {"type": "prometheus", "uid": ""},
"targets": [
{
"expr": "ml_feature_mean",
"legendFormat": "{{feature}}"
}
]
},
{
"id": 8,
"title": "Active Alerts",
"type": "table",
"gridPos": {"h": 7, "w": 24, "x": 0, "y": 21},
"datasource": {"type": "prometheus", "uid": ""},
"targets": [
{
"expr": "ALERTS{alertstate=\"firing\"}",
"format": "table",
"instant": true
}
]
}
],
"time": {"from": "now-15m", "to": "now"},
"refresh": "5s",
"schemaVersion": 39
}What do these 8 panels show?
- Panel 1 (Request Rate) tracks prediction throughput per second, broken down by model version.
- Panel 2 (Latency) overlays p95 and p50 latency using your recording rules. You can spot tail latency spikes instantly.
- Panel 3 (Class Distribution) shows prediction balance across classes. A sudden shift to one class might mean data issues.
- Panel 4 (Drift Score) plots the KS statistic per feature with threshold markers at 0.2 (warning) and 0.3 (critical).
- Panel 5 (Confidence) is a gauge showing the most recent prediction confidence. Red below 0.6, green above 0.8.
- Panel 6 (Error Rate) displays the percentage of failed requests using your error rate recording rule.
- Panel 7 (Feature Mean) tracks rolling averages of input features. Useful for spotting gradual distribution shifts.
- Panel 8 (Active Alerts) is a table of currently firing alerts pulled directly from Prometheus.
- Save grafana/dashboards/ml-monitoring.json.
JSON syntax error when Grafana loads?
JSON is strict about commas and brackets. A trailing comma after the last item in an array or object will cause a parse failure. Double-check that there is no comma after the last panel object in the panels array.
help me validate my Grafana dashboard JSON file for syntax errors
✔️ Awesome, I've got everything!
Great. Make sure you have saved all three files before proceeding.
ⓧ I'd like to double check the full code
Here is the complete datasource provisioning file:
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
access: proxy
url: http://prometheus:9090
isDefault: true
editable: false
jsonData:
timeInterval: "10s"Here is the complete dashboard provisioning file:
apiVersion: 1
providers:
- name: "ML Monitoring"
orgId: 1
folder: "ML Monitoring"
type: file
disableDeletion: false
updateIntervalSeconds: 10
allowUiUpdates: true
options:
path: /var/lib/grafana/dashboards
foldersFromFilesStructure: falseHere is the complete dashboard JSON:
{
"annotations": {"list": []},
"editable": true,
"id": null,
"title": "ML Model Monitoring",
"uid": "ml-monitoring-v1",
"panels": [
{
"id": 1,
"title": "Prediction Request Rate (req/s)",
"type": "timeseries",
"gridPos": {"h": 7, "w": 12, "x": 0, "y": 0},
"datasource": {"type": "prometheus", "uid": ""},
"targets": [
{
"expr": "sum(rate(ml_predictions_total[5m])) by (model_version)",
"legendFormat": "{{model_version}}"
}
],
"fieldConfig": {"defaults": {"unit": "reqps"}}
},
{
"id": 2,
"title": "Prediction Latency (p95 / p50)",
"type": "timeseries",
"gridPos": {"h": 7, "w": 12, "x": 12, "y": 0},
"datasource": {"type": "prometheus", "uid": ""},
"targets": [
{
"expr": "ml:prediction_latency_p95:5m",
"legendFormat": "p95 {{model_version}}"
},
{
"expr": "ml:prediction_latency_p50:5m",
"legendFormat": "p50 {{model_version}}"
}
],
"fieldConfig": {"defaults": {"unit": "s"}}
},
{
"id": 3,
"title": "Prediction Class Distribution",
"type": "timeseries",
"gridPos": {"h": 7, "w": 12, "x": 0, "y": 7},
"datasource": {"type": "prometheus", "uid": ""},
"targets": [
{
"expr": "sum(rate(ml_predictions_total[5m])) by (predicted_class)",
"legendFormat": "class {{predicted_class}}"
}
]
},
{
"id": 4,
"title": "Data Drift Score (KS Statistic)",
"type": "timeseries",
"gridPos": {"h": 7, "w": 12, "x": 12, "y": 7},
"datasource": {"type": "prometheus", "uid": ""},
"targets": [
{
"expr": "ml_data_drift_score",
"legendFormat": "{{feature}} ({{model_version}})"
}
],
"fieldConfig": {
"defaults": {
"max": 1,
"min": 0,
"thresholds": {
"mode": "absolute",
"steps": [
{"color": "green", "value": null},
{"color": "yellow", "value": 0.2},
{"color": "red", "value": 0.3}
]
},
"custom": {"thresholdsStyle": {"mode": "line"}}
}
}
},
{
"id": 5,
"title": "Prediction Confidence",
"type": "gauge",
"gridPos": {"h": 7, "w": 6, "x": 0, "y": 14},
"datasource": {"type": "prometheus", "uid": ""},
"targets": [
{
"expr": "ml_prediction_confidence",
"legendFormat": "{{model_version}}"
}
],
"fieldConfig": {
"defaults": {
"min": 0,
"max": 1,
"thresholds": {
"mode": "absolute",
"steps": [
{"color": "red", "value": null},
{"color": "yellow", "value": 0.6},
{"color": "green", "value": 0.8}
]
}
}
}
},
{
"id": 6,
"title": "Error Rate (%)",
"type": "stat",
"gridPos": {"h": 7, "w": 6, "x": 6, "y": 14},
"datasource": {"type": "prometheus", "uid": ""},
"targets": [
{
"expr": "ml:error_rate:5m * 100",
"legendFormat": "{{model_version}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "percent",
"thresholds": {
"mode": "absolute",
"steps": [
{"color": "green", "value": null},
{"color": "yellow", "value": 2},
{"color": "red", "value": 5}
]
}
}
}
},
{
"id": 7,
"title": "Feature Mean (Rolling Window)",
"type": "timeseries",
"gridPos": {"h": 7, "w": 12, "x": 12, "y": 14},
"datasource": {"type": "prometheus", "uid": ""},
"targets": [
{
"expr": "ml_feature_mean",
"legendFormat": "{{feature}}"
}
]
},
{
"id": 8,
"title": "Active Alerts",
"type": "table",
"gridPos": {"h": 7, "w": 24, "x": 0, "y": 21},
"datasource": {"type": "prometheus", "uid": ""},
"targets": [
{
"expr": "ALERTS{alertstate=\"firing\"}",
"format": "table",
"instant": true
}
]
}
],
"time": {"from": "now-15m", "to": "now"},
"refresh": "5s",
"schemaVersion": 39
}Launch the full stack and verify all services
Every configuration file is in place. Your docker-compose.yml already references the Grafana provisioning paths you just created. Time to bring the entire 4-service stack online.
- Launch all services by running:
docker compose up -dWhat does this command do?
docker compose up -d builds the model-server image from your Dockerfile, pulls the Prometheus, Grafana, and Alertmanager images, then starts all 4 containers in detached mode. The depends_on rules ensure Prometheus waits for the model-server health check to pass before starting.
This may take a minute or two on first run while Docker pulls the images and installs Python dependencies in the build stage.
Once the command finishes, you need to verify each service is healthy.
- Check the model server by running:
curl http://localhost:8000/healthYou should see a JSON response containing "status": "healthy" and "model_version": "v1".
- Check Prometheus by running:
curl http://localhost:9090/-/readyYou should see a message indicating Prometheus is ready to receive traffic.
- Check Grafana by running:
curl http://localhost:3000/api/healthYou should see a JSON response with "database": "ok".
- Check Alertmanager by running:
curl http://localhost:9093/-/readyYou should see a response confirming Alertmanager is ready.
Open Grafana in your browser
Visit http://localhost:3000 in your browser. Because you configured anonymous auth with Admin role, no login is required. Your ML Model Monitoring dashboard loads automatically as the home dashboard.
Service not starting or health check failing?
Run docker compose logs model-server to check for Python import errors or missing files. The most common issue is a typo in a file path.
If Grafana shows no datasource, verify that your grafana/provisioning directory matches the volume mount in docker-compose.yml exactly. The mount is ./grafana/provisioning:/etc/grafana/provisioning.
help me debug why my Docker Compose services are not starting or failing health checks
Your entire observability stack is live. Prometheus is scraping your model server, Grafana is visualizing the metrics, and Alertmanager is standing by. Next up, you will generate traffic to watch the dashboards come alive and trigger real alerts.
Simulate Traffic and Validate the Alerting Pipeline
Your monitoring stack is deployed and configured, but right now it's watching an empty void. A monitoring system only proves its value when it reacts to real problems.
In this step, you'll write a traffic generator that simulates three distinct failure scenarios: normal operation, data drift, and malformed requests. You'll watch your alerts transition through their lifecycle and confirm that Alertmanager routes them correctly.
In this step, get ready to:
- Write a multi-phase traffic generator script that simulates normal, drifted, and error traffic.
- Run the generator and observe alert transitions in Prometheus and Grafana.
- Verify end-to-end alerting with severity routing, inhibition, and grouping in Alertmanager.
Write the multi-phase traffic generator script
The traffic generator needs to produce three distinct signal patterns so each of your alert rules gets exercised independently.
- Create the traffic generator file by running:
touch scripts/generate_traffic.py- Confirm the file exists by running:
ls scripts/- You should see generate_traffic.py in the output.
- In scripts/generate_traffic.py, add the imports, feature range constants, and sample generation functions:
import time
import random
import sys
import requests
import numpy as np
BASE_URL = "http://localhost:8000"
# Normal Iris feature ranges (sepal length, sepal width, petal length, petal width)
NORMAL_RANGES = [
(4.3, 7.9),
(2.0, 4.4),
(1.0, 6.9),
(0.1, 2.5),
]
# Shifted ranges far outside training distribution to trigger drift
DRIFTED_RANGES = [
(10.0, 15.0),
(5.0, 8.0),
(8.0, 12.0),
(3.0, 5.0),
]
def generate_normal_sample():
return [random.uniform(lo, hi) for lo, hi in NORMAL_RANGES]
def generate_drifted_sample():
return [random.uniform(lo, hi) for lo, hi in DRIFTED_RANGES]What does this code do?
NORMAL_RANGES defines feature value bounds that match the Iris training distribution. Samples drawn from these ranges look like data the model was trained on.
DRIFTED_RANGES defines values far outside the training distribution. When the KS test compares these against the reference data, the drift score will exceed the 0.3 threshold and fire the DataDriftDetected alert.
- Below the sample generators, add the HTTP helper functions that send requests and display progress:
def send_prediction(features):
try:
response = requests.post(
f"{BASE_URL}/predict",
json={"features": features},
timeout=5,
)
if response.status_code == 200:
return response.json()
return {"error": response.status_code}
except requests.exceptions.ConnectionError:
return {"error": "connection_failed"}
except Exception as e:
return {"error": str(e)}
def send_bad_request():
# Sends only 2 features instead of 4 to trigger validation error
try:
response = requests.post(
f"{BASE_URL}/predict",
json={"features": [1.0, 2.0]},
timeout=5,
)
return {"error": response.status_code}
except Exception as e:
return {"error": str(e)}
def print_status(phase, iteration, total, result):
bar_length = 30
progress = int(bar_length * iteration / total)
bar = "=" * progress + "-" * (bar_length - progress)
if "error" not in result:
status = f"class={result['prediction']} conf={result['confidence']:.2f}"
else:
status = f"error={result['error']}"
sys.stdout.write(f"\r [{bar}] {iteration}/{total}s | {status} ")
sys.stdout.flush()What do these helpers do?
- send_prediction() posts a valid 4-feature payload to /predict and returns the JSON response or an error dict.
- send_bad_request() deliberately sends only 2 features. The model server rejects this with a 422 and increments the ml_prediction_errors_total counter.
- print_status() renders a live progress bar so you can watch each phase execute in real time.
- Finally, add the main() function that orchestrates all three traffic phases:
def main():
print("=" * 60)
print(" ML Monitoring Traffic Generator")
print("=" * 60)
# Check service health
print("\nChecking service health...")
try:
resp = requests.get(f"{BASE_URL}/health", timeout=5)
health = resp.json()
print(f" Service: {health['status']}")
print(f" Model: {health['model_version']} ({health['model_algorithm']})")
except Exception as e:
print(f" ERROR: Cannot reach service at {BASE_URL}: {e}")
print(" Make sure the stack is running: docker compose up -d")
sys.exit(1)
# Phase 1: Normal traffic
print(f"\nPhase 1: Normal traffic (60 seconds)")
print(" Establishes baseline metrics...")
for i in range(1, 61):
result = send_prediction(generate_normal_sample())
print_status("normal", i, 60, result)
time.sleep(1)
print()
# Phase 2: Drifted traffic
print(f"\nPhase 2: Drifted traffic (60 seconds)")
print(" Sending out-of-distribution data to trigger drift alerts...")
for i in range(1, 61):
result = send_prediction(generate_drifted_sample())
print_status("drifted", i, 60, result)
time.sleep(1)
print()
# Phase 3: Error traffic
print(f"\nPhase 3: Error traffic (20 seconds)")
print(" Sending malformed requests to trigger error rate alerts...")
for i in range(1, 21):
result = send_bad_request()
print_status("errors", i, 20, result)
time.sleep(1)
print()
# Summary
print("\n" + "=" * 60)
print(" Traffic generation complete!")
print("=" * 60)
print(f"\n Grafana dashboard: http://localhost:3000")
print(f" Prometheus alerts: http://localhost:9090/alerts")
print(f" Alertmanager: http://localhost:9093")
print(f"\n Expected alerts after ~2 minutes:")
print(f" - DataDriftDetected (critical)")
print(f" - LowPredictionConfidence (warning, inhibited by drift)")
print(f" - HighErrorRate (critical)")
if __name__ == "__main__":
main()How do the three phases map to alerts?
- Phase 1 (60s) sends in-distribution data to establish a healthy baseline. No alerts should fire during this window.
- Phase 2 (60s) sends out-of-distribution data. After the rolling window fills (50 requests), the KS drift score exceeds 0.3, triggering DataDriftDetected (critical) and LowPredictionConfidence (warning, which gets inhibited by the critical drift alert).
- Phase 3 (20s) sends malformed 2-feature requests. The error rate exceeds 5%, firing HighErrorRate (critical).
- Save scripts/generate_traffic.py.
✔️ Awesome, I've got everything!
Great. Double check you've saved the file before moving on.
ⓧ I'd like to double check the full code
import time
import random
import sys
import requests
import numpy as np
BASE_URL = "http://localhost:8000"
NORMAL_RANGES = [
(4.3, 7.9),
(2.0, 4.4),
(1.0, 6.9),
(0.1, 2.5),
]
DRIFTED_RANGES = [
(10.0, 15.0),
(5.0, 8.0),
(8.0, 12.0),
(3.0, 5.0),
]
def generate_normal_sample():
return [random.uniform(lo, hi) for lo, hi in NORMAL_RANGES]
def generate_drifted_sample():
return [random.uniform(lo, hi) for lo, hi in DRIFTED_RANGES]
def send_prediction(features):
try:
response = requests.post(
f"{BASE_URL}/predict",
json={"features": features},
timeout=5,
)
if response.status_code == 200:
return response.json()
return {"error": response.status_code}
except requests.exceptions.ConnectionError:
return {"error": "connection_failed"}
except Exception as e:
return {"error": str(e)}
def send_bad_request():
try:
response = requests.post(
f"{BASE_URL}/predict",
json={"features": [1.0, 2.0]},
timeout=5,
)
return {"error": response.status_code}
except Exception as e:
return {"error": str(e)}
def print_status(phase, iteration, total, result):
bar_length = 30
progress = int(bar_length * iteration / total)
bar = "=" * progress + "-" * (bar_length - progress)
if "error" not in result:
status = f"class={result['prediction']} conf={result['confidence']:.2f}"
else:
status = f"error={result['error']}"
sys.stdout.write(f"\r [{bar}] {iteration}/{total}s | {status} ")
sys.stdout.flush()
def main():
print("=" * 60)
print(" ML Monitoring Traffic Generator")
print("=" * 60)
# Check service health
print("\nChecking service health...")
try:
resp = requests.get(f"{BASE_URL}/health", timeout=5)
health = resp.json()
print(f" Service: {health['status']}")
print(f" Model: {health['model_version']} ({health['model_algorithm']})")
except Exception as e:
print(f" ERROR: Cannot reach service at {BASE_URL}: {e}")
print(" Make sure the stack is running: docker compose up -d")
sys.exit(1)
# Phase 1: Normal traffic
print(f"\nPhase 1: Normal traffic (60 seconds)")
print(" Establishes baseline metrics...")
for i in range(1, 61):
result = send_prediction(generate_normal_sample())
print_status("normal", i, 60, result)
time.sleep(1)
print()
# Phase 2: Drifted traffic
print(f"\nPhase 2: Drifted traffic (60 seconds)")
print(" Sending out-of-distribution data to trigger drift alerts...")
for i in range(1, 61):
result = send_prediction(generate_drifted_sample())
print_status("drifted", i, 60, result)
time.sleep(1)
print()
# Phase 3: Error traffic
print(f"\nPhase 3: Error traffic (20 seconds)")
print(" Sending malformed requests to trigger error rate alerts...")
for i in range(1, 21):
result = send_bad_request()
print_status("errors", i, 20, result)
time.sleep(1)
print()
# Summary
print("\n" + "=" * 60)
print(" Traffic generation complete!")
print("=" * 60)
print(f"\n Grafana dashboard: http://localhost:3000")
print(f" Prometheus alerts: http://localhost:9090/alerts")
print(f" Alertmanager: http://localhost:9093")
print(f"\n Expected alerts after ~2 minutes:")
print(f" - DataDriftDetected (critical)")
print(f" - LowPredictionConfidence (warning, inhibited by drift)")
print(f" - HighErrorRate (critical)")
if __name__ == "__main__":
main()Run the generator and observe alerts in Grafana
The script takes about 2.5 minutes to complete all three phases. While it runs, you'll watch the Grafana dashboard react in real time.
- Open your Grafana dashboard in your browser by navigating to:
http://localhost:3000- You should see the ML Model Monitoring dashboard with 8 panels, all showing no data yet.
- In a separate terminal tab (keep Grafana open in your browser), run the traffic generator:
python scripts/generate_traffic.pyWhat should I see in the terminal?
The script first checks the model server's health, then begins Phase 1 with a live progress bar. You'll see prediction class and confidence values updating each second.
During Phase 2, confidence values will drop noticeably as the model receives out-of-distribution inputs. During Phase 3, you'll see 422 error codes for every request.
- While Phase 1 runs, switch to your Grafana browser tab. You should see the Prediction Request Rate panel begin climbing to approximately 1 req/s.
- Watch the Data Drift Score panel during Phase 2. After about 50 seconds (when the rolling window fills), drift scores will jump above the red 0.3 threshold line.
- Watch the Prediction Confidence gauge drop during Phase 2 as the model is less certain about out-of-distribution inputs.
- Watch the Error Rate (%) panel spike during Phase 3 as malformed requests pile up.
Dashboard auto-refreshes every 5 seconds
The dashboard is configured with a 5-second refresh interval and a 15-minute time window. You don't need to manually reload the page.
- After the script finishes, open the Prometheus alerts page to confirm alert state transitions:
http://localhost:9090/alerts- Look for DataDriftDetected in the ml_data_quality_alerts group. It should show a state of firing (red).
- The HighErrorRate alert in the ml_service_health_alerts group should also be firing.
Why do alerts take a minute or two to fire?
Each alert rule has a for duration (e.g., 1m for DataDriftDetected). The condition must be true continuously for that duration before the alert transitions from pending to firing. This prevents flapping on momentary spikes.
Not seeing any alerts firing?
- Wait at least 2 minutes after the script finishes. The DataDriftDetected alert has a 1-minute for duration, and Prometheus evaluates rules every 15 seconds.
- Check that Prometheus is scraping successfully. On the Prometheus page at localhost:9090/targets, the ml-model-server target should show state UP.
- If the traffic generator printed connection errors, make sure your Docker Compose stack is running with docker compose ps and all 4 services are healthy.
Help me debug why my Prometheus alerts aren't firing after running the traffic generator.
Verify end-to-end alerting in Alertmanager
Prometheus fires alerts, but Alertmanager is responsible for routing, grouping, and deduplicating them. This is where your routing tree, inhibition rules, and receiver configuration prove they work.
- Open the Alertmanager UI in your browser:
http://localhost:9093- You should see alert groups displayed on the page. Look for a group with severity=critical containing the DataDriftDetected alert.
- Verify the grouping labels. Alerts are grouped by alertname and model_version as configured in your routing tree.
- Check that HighErrorRate also appears as a separate group with severity=critical.
What about the LowPredictionConfidence alert?
You should NOT see LowPredictionConfidence in the Alertmanager UI. Your inhibition rule suppresses it: when DataDriftDetected (critical, component=data-quality) is firing, it inhibits LowPredictionConfidence (warning, component=model-quality) for the same model_version.
This makes operational sense. Drift causes low confidence, so there's no point alerting on both. The critical alert is the root cause.
- Verify the routing worked by checking receiver assignment. Critical alerts should be routed to the webhook-critical receiver with a group_wait of 10 seconds (faster notification for critical issues).
- Click on any alert group in the Alertmanager UI to expand its details. You'll see the full annotation labels including summary, description, and runbook fields that you defined in your alert rules.
How does the routing tree decide where alerts go?
Your routing tree processes alerts top-down. An alert with severity=critical matches the first child route and goes to webhook-critical with continue: false (stops checking further routes). A warning matches the second route and goes to webhook-warning. Anything else falls through to the default receiver webhook-info.
- Return to the Grafana dashboard at localhost:3000. The Active Alerts table panel (bottom of the dashboard) should now display the firing alerts with their labels and severity.
Alertmanager UI shows no alerts?
- Confirm that Prometheus can reach Alertmanager. In Prometheus at localhost:9090/alerts, alerts should show as firing there first. If they do, the issue is the connection between Prometheus and Alertmanager.
- Check Alertmanager container logs by running docker compose logs alertmanager to look for configuration errors.
- Verify the alerting section in prometheus/prometheus.yml points to alertmanager:9093 (the Docker service name, not localhost).
Help me figure out why Alertmanager isn't receiving alerts from Prometheus.
You've just validated a production-grade alerting pipeline end-to-end. Your monitoring stack detects drift, routes critical alerts with low latency, suppresses redundant warnings through inhibition, and groups related alerts for clean operator workflows.
Secret mission
Model Version Comparison Metrics
Your monitoring stack tracks a single model version. In production, teams deploy multiple model versions side by side to compare performance before fully rolling out a new release. Can you add multi-model serving with a version query parameter and build Grafana comparison panels that reveal performance differences between a RandomForest and a simpler DecisionTree?
Clean Up Your Resources
Clean Up Your Resources
Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally via Docker containers, so there are no ongoing cloud costs.
Resources you used:
- Docker containers: model-server, prometheus, grafana, alertmanager.
- Docker volumes: prometheus-data (Prometheus TSDB data) and grafana-data (Grafana state).
- Python virtual environment and project files on your Desktop.
Keep everything running
No action needed. Choose this if you're still actively experimenting with the monitoring stack.
- Your Docker Compose stack stays up and accessible at its usual ports.
- Grafana remains at localhost:3000, Prometheus at localhost:9090, and Alertmanager at localhost:9093.
- Everything is local Docker, so there is no financial cost to keeping it running.
- The containers will restart automatically after a system reboot thanks to the unless-stopped restart policy.
Pause - I'll come back to this later
Shut down running containers to free up memory and CPU, while keeping your data intact in Docker volumes.
- Stop all containers by running this command from your project directory:
docker compose stop- This pauses the containers without removing them or their volumes.
- Your Prometheus TSDB data and Grafana dashboard state are preserved.
- To resume later, navigate back to your project directory and run:
docker compose startDelete - I don't want to use this again
Remove all Docker containers, volumes, and project files to start completely fresh.
- From your project directory, remove all containers, volumes, and the Docker network by running:
docker compose down -v- This removes the model-server, prometheus, grafana, and alertmanager containers along with the prometheus-data and grafana-data volumes.
- Deactivate your Python virtual environment by running:
deactivate- Move up one directory and delete the entire project folder by running:
cd ~/Desktop
rm -rf ml-monitoringOn Windows, use this command instead:
cd ~\Desktop
rmdir /s /q ml-monitoring- This frees all disk space used by the project files, model artifacts, and configuration.
Nice Work!
Nice Work!
You just built a production-grade ML monitoring stack from scratch. Your local environment now runs a complete observability pipeline where drifted model inputs trigger severity-based alerts routed through Alertmanager with inhibition and grouping.
You've learned how to:
- Built a production FastAPI inference service instrumented with 7 custom Prometheus metrics covering latency, throughput, errors, drift, confidence, and feature statistics.
- Deployed a 4-service observability stack via Docker Compose with Prometheus recording rules, 6 alert rules across 3 severity groups, and a Grafana dashboard provisioned entirely as code.
- Simulated multi-phase traffic (normal, drifted, and error-inducing) and validated the complete alerting pipeline end-to-end, from drift detection through severity-based routing with inhibition rules.
- Secret Mission: Added model version routing with a query parameter and built comparison dashboards showing performance differences across model versions.
Ready to quiz yourself?