GitOps Canary Deployments for LLM Services

Introduction

30 Second Summary

Every time a team updates a machine learning model in production, they face a nerve-wracking question: will this new version break things for real users? The safest teams never flip a switch all at once. They route a sliver of traffic to the new version, watch the metrics, and let automation decide whether to proceed or pull back.

In this project, you will build a complete MLOps pipeline on your local machine that automates this entire workflow. A single git push triggers a container build, a GitOps sync to your Kubernetes cluster, and a metrics-driven canary release that automatically promotes or rolls back your model service.

What You'll Build

A production-grade MLOps pipeline that deploys a FastAPI-based LLM inference service through automated canary releases, with Prometheus analysis deciding whether each new version is safe to promote.

By the end of this project, you'll have:

• A full GitOps pipeline where pushing code to GitHub automatically builds a container image, syncs it to your Kubernetes cluster via ArgoCD, and performs a canary release with real traffic splitting.
• An Argo Rollouts canary deployment that routes 10% of traffic to new versions, with Prometheus-driven analysis automatically rolling back if metrics degrade.
• A Grafana dashboard with alerting rules visualizing inference latency, error rate, and prediction quality, backed by a PostgreSQL prediction log for long-term model drift analysis.
• Secret Mission: add a chaos injection mode that deliberately degrades responses to trigger and observe the automated rollback in action.

Are there any prerequisites?

This project assumes familiarity with Kubernetes concepts (deployments, services, ingress), basic Terraform usage, and a standard Git/GitHub workflow.

You'll need Rancher Desktop, Docker, Terraform, the ArgoCD CLI, Ollama, and a GitHub account already available on your Linux Ubuntu machine.

Before We Start

This project deploys a full MLOps platform stack on your local machine. That means six or more services running simultaneously in your Kubernetes cluster, all coordinated by GitOps tooling.

Before writing any application code, you need to confirm your CLI tools work, configure Ollama for network access from pods, and install a few remaining utilities. This step gets your environment production-ready.

In this step, get ready to:

• Verify all CLI tools and confirm your k3s cluster is running with enough resources.
• Configure Ollama to listen on all network interfaces so pods can reach it.
• Disable Traefik and install the Argo Rollouts kubectl plugin and the hey load tester.

Verify CLI tools and cluster resources

The platform stack you are about to deploy depends on several CLI tools working together. Confirming each one now prevents cryptic errors later.

• Open a terminal by pressing the Super key (Ubuntu) and typing Terminal, then pressing Enter.
• Run each of these version checks to confirm your tools are installed:

kubectl version --client
helm version
terraform version
docker version
argocd version --client
ollama --version

What should I see?

Each command should print a version number without errors. If any command returns command not found, that tool is not installed or not in your PATH. Revisit the tool's official installation docs before continuing.

The full platform stack (ArgoCD, Argo Rollouts, Prometheus, Grafana, NGINX Ingress, PostgreSQL, plus your app) needs significant memory. You need at least 8GB allocated to Rancher Desktop.

• Open Rancher Desktop.
• Click Preferences (gear icon in the top bar).
• Select Virtual Machine in the sidebar.
• Confirm that Memory is set to at least 8192 MB (8 GB). If it is lower, increase it and click Apply.

• Verify your k3s cluster is running by running this command:

kubectl get nodes

You should see a single node with status Ready. This confirms Rancher Desktop's k3s cluster is healthy.

Node not showing Ready?

If the node shows NotReady, Rancher Desktop may still be starting up. Wait 30 seconds and run the command again.

If no nodes appear at all, open Rancher Desktop and confirm that Kubernetes is enabled in Preferences > Kubernetes.

Help me troubleshoot my Rancher Desktop k3s cluster not showing Ready.

Configure Ollama to listen on all interfaces

By default, Ollama only listens on 127.0.0.1 (localhost). Your inference service will run inside Kubernetes pods, which are on a different network namespace. Pods trying to reach Ollama on localhost will get a connection refused error.

You need to bind Ollama to 0.0.0.0 so it accepts connections from any interface, including the pod network.

• Create the systemd override directory by running this command:

sudo mkdir -p /etc/systemd/system/ollama.service.d

• Create the override configuration file by running this command:

sudo tee /etc/systemd/system/ollama.service.d/override.conf <<EOF
[Service]
Environment="OLLAMA_HOST=0.0.0.0"
EOF

What does this command do?

This creates a systemd override file that sets the OLLAMA_HOST environment variable to 0.0.0.0. When Ollama starts, systemd injects this variable, telling Ollama to listen on all network interfaces instead of just localhost.

• Reload systemd and restart Ollama to apply the change by running these commands:

sudo systemctl daemon-reload && sudo systemctl restart ollama

• Verify Ollama is listening on all interfaces by running:

ss -tlnp | grep 11434

You should see 0.0.0.0:11434 in the output. If you see 127.0.0.1:11434 instead, the override did not apply correctly.

Now pull a small model that your inference service will use. This keeps inference fast and memory usage low.

• Pull the qwen2:0.5b model by running:

ollama pull qwen2:0.5b

The download may take a minute depending on your connection speed. Once complete, you should see a success message confirming the model is ready.

Seeing connection refused when pulling the model?

If ollama pull fails, check that the Ollama service is running with systemctl status ollama. If it shows inactive or failed, check the override file for typos (the quotes around OLLAMA_HOST=0.0.0.0 must be present).

Help me troubleshoot Ollama not starting after configuring the systemd override.

Disable Traefik and install remaining tools

k3s ships with Traefik as its default ingress controller, which binds ports 80 and 443. You will install NGINX Ingress Controller instead, because Argo Rollouts has well-documented canary traffic routing integration with NGINX.

Why NGINX instead of Traefik?

Argo Rollouts needs to create a second ingress with canary-weight annotations during progressive delivery. The NGINX integration for this is mature and extensively documented. Traefik does support traffic splitting, but the NGINX path has far more community examples and is the recommended approach in the Argo Rollouts docs.

• Disable Traefik in Rancher Desktop by running this command:

rdctl set --kubernetes.options.traefik=false

What does this command do?

The rdctl command configures Rancher Desktop from the terminal. Setting --kubernetes.options.traefik=false tells k3s to stop running Traefik, freeing ports 80 and 443 for NGINX Ingress Controller.

• Restart Rancher Desktop for the change to take effect. Close Rancher Desktop fully, then reopen it.
• Wait for the cluster to come back up (30-60 seconds), then confirm Traefik is gone by running:

kubectl get pods -n kube-system | grep traefik

This should return no results. If you still see Traefik pods, wait another minute and check again.

Now install the Argo Rollouts kubectl plugin. This gives you CLI commands to manage and visualize canary rollouts.

• Download, make executable, and install the plugin by running these commands:

curl -LO https://github.com/argoproj/argo-rollouts/releases/latest/download/kubectl-argo-rollouts-linux-amd64
chmod +x ./kubectl-argo-rollouts-linux-amd64
sudo mv ./kubectl-argo-rollouts-linux-amd64 /usr/local/bin/kubectl-argo-rollouts

What do these commands do?

• curl -LO downloads the pre-built binary from the Argo Rollouts GitHub releases.
• chmod +x makes the binary executable.
• sudo mv moves it into your system PATH so kubectl can discover it as a plugin.

• Verify the plugin is installed by running:

kubectl argo rollouts version

You should see a version number printed. This confirms the plugin is discoverable by kubectl.

Finally, install hey, an HTTP load testing tool. You will use it later to generate traffic during canary releases so Prometheus has real metrics to evaluate.

• Download and install hey by running these commands:

curl -LO https://storage.googleapis.com/hey-releases/hey_linux_amd64
chmod +x hey_linux_amd64
sudo mv hey_linux_amd64 /usr/local/bin/hey

• Verify hey is installed by running:

hey -n 1 https://example.com

You should see a summary of the test request with response time statistics. This confirms hey is working.

Permission denied or command not found?

If you get a permission error, make sure you used sudo when running the mv command. The /usr/local/bin/ directory requires root access to write to.

If the command is not found after installation, confirm the file is in your PATH by running ls /usr/local/bin/hey. If it exists but isn't found, open a new terminal session.

Help me troubleshoot installing the hey load tester or kubectl-argo-rollouts plugin on Linux.

Your environment is ready. Every tool is verified, Ollama is reachable from pod networks, and Traefik is out of the way. Next up, you will use Terraform to bootstrap the entire platform stack in one command.

Bootstrap the Platform with Terraform

With your cluster running and tools in place, you need a way to install the entire observability and deployment platform in a single, repeatable command.

Terraform lets you declare your desired platform state as code. In this step, you will write a Terraform configuration that installs six Helm charts into your cluster and then apply it all at once.

In this step, get ready to:

• Create a Terraform configuration that installs NGINX Ingress, ArgoCD, Argo Rollouts, kube-prometheus-stack, and PostgreSQL.
• Apply the configuration to provision all platform components in one command.
• Verify every component is running and access the ArgoCD, Grafana, and Argo Rollouts dashboards.

Create the Terraform configuration

Your Terraform configuration will live in a dedicated directory with four files, each responsible for a different concern.

• Create a terraform directory inside your project folder by running these commands:

mkdir terraform
cd terraform

• Create the provider declarations file by adding a new file called providers.tf with the following content:

# providers.tf - Declare the Helm and Kubernetes providers

terraform {
  required_providers {
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.12"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.25"
    }
  }
}

# Point both providers at the local kubeconfig (Rancher Desktop k3s)
provider "helm" {
  kubernetes {
    config_path = "~/.kube/config"
  }
}

provider "kubernetes" {
  config_path = "~/.kube/config"
}

What does this file do?

The helm provider lets Terraform install Helm charts directly. The kubernetes provider lets Terraform create raw Kubernetes resources like namespaces.

Both providers read your local ~/.kube/config file, which Rancher Desktop already configured to point at your k3s cluster.

• Create the variables file by adding variables.tf with the following content:

# variables.tf - Chart versions and configuration values

variable "argo_rollouts_chart_version" {
  description = "Argo Rollouts Helm chart version"
  type        = string
  default     = "2.40.9"
}

variable "kube_prometheus_stack_chart_version" {
  description = "kube-prometheus-stack Helm chart version"
  type        = string
  default     = "85.2.0"
}

variable "postgresql_chart_version" {
  description = "Bitnami PostgreSQL Helm chart version"
  type        = string
  default     = "18.6.7"
}

variable "postgresql_password" {
  description = "Password for the predictions database"
  type        = string
  default     = "mlops-pipeline"
}

variable "postgresql_database" {
  description = "Database name for prediction logging"
  type        = string
  default     = "predictions"
}

What does this file do?

Variables pin the Helm chart versions so your configuration is reproducible. The Argo Rollouts chart is pinned to 2.40.9 (appVersion v1.9.0), kube-prometheus-stack to 85.2.0, and PostgreSQL to 18.6.7.

The PostgreSQL password and database name are set here so other services can connect later using these values.

• Create the main configuration file by adding main.tf with the following content:

# main.tf - Platform components installed via Helm

# --- Namespaces ---

resource "kubernetes_namespace" "ingress_nginx" {
  metadata {
    name = "ingress-nginx"
  }
}

resource "kubernetes_namespace" "argocd" {
  metadata {
    name = "argocd"
  }
}

resource "kubernetes_namespace" "argo_rollouts" {
  metadata {
    name = "argo-rollouts"
  }
}

resource "kubernetes_namespace" "monitoring" {
  metadata {
    name = "monitoring"
  }
}

resource "kubernetes_namespace" "postgres" {
  metadata {
    name = "postgres"
  }
}

# --- NGINX Ingress Controller ---

resource "helm_release" "nginx_ingress" {
  name       = "ingress-nginx"
  repository = "https://kubernetes.github.io/ingress-nginx"
  chart      = "ingress-nginx"
  namespace  = kubernetes_namespace.ingress_nginx.metadata[0].name

  # Use LoadBalancer type so k3s ServiceLB binds ports 80/443 to the host
  set {
    name  = "controller.service.type"
    value = "LoadBalancer"
  }
}

# --- ArgoCD ---

resource "helm_release" "argocd" {
  name       = "argocd"
  repository = "https://argoproj.github.io/argo-helm"
  chart      = "argo-cd"
  namespace  = kubernetes_namespace.argocd.metadata[0].name

  # Keep the server as ClusterIP since we port-forward to access it
  set {
    name  = "server.service.type"
    value = "ClusterIP"
  }
}

# --- Argo Rollouts with dashboard ---

resource "helm_release" "argo_rollouts" {
  name       = "argo-rollouts"
  repository = "https://argoproj.github.io/argo-helm"
  chart      = "argo-rollouts"
  version    = var.argo_rollouts_chart_version
  namespace  = kubernetes_namespace.argo_rollouts.metadata[0].name

  # Enable the Argo Rollouts dashboard for visualising rollout states
  set {
    name  = "dashboard.enabled"
    value = "true"
  }
}

# --- kube-prometheus-stack (Prometheus + Grafana) ---

resource "helm_release" "kube_prometheus_stack" {
  name       = "prometheus"
  repository = "https://prometheus-community.github.io/helm-charts"
  chart      = "kube-prometheus-stack"
  version    = var.kube_prometheus_stack_chart_version
  namespace  = kubernetes_namespace.monitoring.metadata[0].name

  # Allow Prometheus to discover ServiceMonitors in ALL namespaces
  set {
    name  = "prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues"
    value = "false"
  }

  # Set a known Grafana password for local access
  set {
    name  = "grafana.adminPassword"
    value = "admin"
  }
}

# --- PostgreSQL (Bitnami) for prediction logging ---

resource "helm_release" "postgresql" {
  name       = "postgresql"
  repository = "oci://registry-1.docker.io/bitnamicharts"
  chart      = "postgresql"
  version    = var.postgresql_chart_version
  namespace  = kubernetes_namespace.postgres.metadata[0].name

  # Database credentials for the inference service
  set {
    name  = "auth.password"
    value = var.postgresql_password
  }

  set {
    name  = "auth.database"
    value = var.postgresql_database
  }
}

What does this configuration install?

• NGINX Ingress Controller handles external traffic routing and is the integration point for Argo Rollouts canary traffic splitting.
• ArgoCD watches your Git repository and automatically syncs Kubernetes manifests to the cluster.
• Argo Rollouts replaces standard Deployments with progressive delivery strategies like canary releases.
• kube-prometheus-stack installs Prometheus (metrics collection) and Grafana (dashboards). The serviceMonitorSelectorNilUsesHelmValues: false setting is critical. It tells Prometheus to pick up ServiceMonitor resources from any namespace, not just ones matching the Helm release labels.
• PostgreSQL stores every prediction the inference service makes so you can analyze model drift over time.

• Create the outputs file by adding outputs.tf with the following content:

# outputs.tf - Useful commands for accessing platform services

output "argocd_password_command" {
  description = "Command to retrieve the ArgoCD admin password"
  value       = "kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath=\"{.data.password}\" | base64 -d"
}

output "argocd_port_forward" {
  description = "Port-forward command for ArgoCD UI"
  value       = "kubectl port-forward svc/argocd-server -n argocd 8080:443"
}

output "grafana_port_forward" {
  description = "Port-forward command for Grafana UI"
  value       = "kubectl port-forward svc/prometheus-grafana -n monitoring 3000:80"
}

output "argo_rollouts_dashboard_port_forward" {
  description = "Port-forward command for Argo Rollouts dashboard"
  value       = "kubectl port-forward svc/argo-rollouts-dashboard -n argo-rollouts 3100:3100"
}

What does this file do?

Outputs print the exact commands you need to access each service after Terraform finishes. You will use these shortly to verify everything is working.

• Save all four files.
• Confirm the directory structure by running:

ls

You should see main.tf, outputs.tf, providers.tf, and variables.tf listed.

Missing a file?

Double-check you are inside the terraform directory. If you see your project root files instead, run cd terraform first.

Help me debug my Terraform directory structure.

✔️ Awesome, I've got everything!

Great. Make sure all files are saved before moving on.

ⓧ I'd like to double check the full code

Here is the complete content of each file for comparison.

# providers.tf - Declare the Helm and Kubernetes providers

terraform {
  required_providers {
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.12"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.25"
    }
  }
}

# Point both providers at the local kubeconfig (Rancher Desktop k3s)
provider "helm" {
  kubernetes {
    config_path = "~/.kube/config"
  }
}

provider "kubernetes" {
  config_path = "~/.kube/config"
}

# variables.tf - Chart versions and configuration values

variable "argo_rollouts_chart_version" {
  description = "Argo Rollouts Helm chart version"
  type        = string
  default     = "2.40.9"
}

variable "kube_prometheus_stack_chart_version" {
  description = "kube-prometheus-stack Helm chart version"
  type        = string
  default     = "85.2.0"
}

variable "postgresql_chart_version" {
  description = "Bitnami PostgreSQL Helm chart version"
  type        = string
  default     = "18.6.7"
}

variable "postgresql_password" {
  description = "Password for the predictions database"
  type        = string
  default     = "mlops-pipeline"
}

variable "postgresql_database" {
  description = "Database name for prediction logging"
  type        = string
  default     = "predictions"
}

# main.tf - Platform components installed via Helm

# --- Namespaces ---

resource "kubernetes_namespace" "ingress_nginx" {
  metadata {
    name = "ingress-nginx"
  }
}

resource "kubernetes_namespace" "argocd" {
  metadata {
    name = "argocd"
  }
}

resource "kubernetes_namespace" "argo_rollouts" {
  metadata {
    name = "argo-rollouts"
  }
}

resource "kubernetes_namespace" "monitoring" {
  metadata {
    name = "monitoring"
  }
}

resource "kubernetes_namespace" "postgres" {
  metadata {
    name = "postgres"
  }
}

# --- NGINX Ingress Controller ---

resource "helm_release" "nginx_ingress" {
  name       = "ingress-nginx"
  repository = "https://kubernetes.github.io/ingress-nginx"
  chart      = "ingress-nginx"
  namespace  = kubernetes_namespace.ingress_nginx.metadata[0].name

  # Use LoadBalancer type so k3s ServiceLB binds ports 80/443 to the host
  set {
    name  = "controller.service.type"
    value = "LoadBalancer"
  }
}

# --- ArgoCD ---

resource "helm_release" "argocd" {
  name       = "argocd"
  repository = "https://argoproj.github.io/argo-helm"
  chart      = "argo-cd"
  namespace  = kubernetes_namespace.argocd.metadata[0].name

  # Keep the server as ClusterIP since we port-forward to access it
  set {
    name  = "server.service.type"
    value = "ClusterIP"
  }
}

# --- Argo Rollouts with dashboard ---

resource "helm_release" "argo_rollouts" {
  name       = "argo-rollouts"
  repository = "https://argoproj.github.io/argo-helm"
  chart      = "argo-rollouts"
  version    = var.argo_rollouts_chart_version
  namespace  = kubernetes_namespace.argo_rollouts.metadata[0].name

  # Enable the Argo Rollouts dashboard for visualising rollout states
  set {
    name  = "dashboard.enabled"
    value = "true"
  }
}

# --- kube-prometheus-stack (Prometheus + Grafana) ---

resource "helm_release" "kube_prometheus_stack" {
  name       = "prometheus"
  repository = "https://prometheus-community.github.io/helm-charts"
  chart      = "kube-prometheus-stack"
  version    = var.kube_prometheus_stack_chart_version
  namespace  = kubernetes_namespace.monitoring.metadata[0].name

  # Allow Prometheus to discover ServiceMonitors in ALL namespaces
  set {
    name  = "prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues"
    value = "false"
  }

  # Set a known Grafana password for local access
  set {
    name  = "grafana.adminPassword"
    value = "admin"
  }
}

# --- PostgreSQL (Bitnami) for prediction logging ---

resource "helm_release" "postgresql" {
  name       = "postgresql"
  repository = "oci://registry-1.docker.io/bitnamicharts"
  chart      = "postgresql"
  version    = var.postgresql_chart_version
  namespace  = kubernetes_namespace.postgres.metadata[0].name

  # Database credentials for the inference service
  set {
    name  = "auth.password"
    value = var.postgresql_password
  }

  set {
    name  = "auth.database"
    value = var.postgresql_database
  }
}

# outputs.tf - Useful commands for accessing platform services

output "argocd_password_command" {
  description = "Command to retrieve the ArgoCD admin password"
  value       = "kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath=\"{.data.password}\" | base64 -d"
}

output "argocd_port_forward" {
  description = "Port-forward command for ArgoCD UI"
  value       = "kubectl port-forward svc/argocd-server -n argocd 8080:443"
}

output "grafana_port_forward" {
  description = "Port-forward command for Grafana UI"
  value       = "kubectl port-forward svc/prometheus-grafana -n monitoring 3000:80"
}

output "argo_rollouts_dashboard_port_forward" {
  description = "Port-forward command for Argo Rollouts dashboard"
  value       = "kubectl port-forward svc/argo-rollouts-dashboard -n argo-rollouts 3100:3100"
}

Apply the Terraform configuration

With all files in place, you can now provision the entire platform stack in two commands.

• Initialize Terraform by running:

terraform init

You should see output confirming the Helm and Kubernetes providers were downloaded successfully.

Seeing a provider download error?

Make sure you have internet connectivity. Terraform needs to download the provider plugins on first init. If you see a permissions error, check that your user has write access to the current directory.

Help me troubleshoot terraform init failing.

• Apply the configuration to install all platform components by running:

terraform apply -auto-approve

How long does this take?

This command creates 5 namespaces and installs 5 Helm charts. Expect it to take 3-5 minutes as container images are pulled to your machine for the first time. The kube-prometheus-stack chart is the largest and takes the longest.

When it finishes, you should see Apply complete! Resources: 10 added, 0 changed, 0 destroyed. along with the output values showing the port-forward commands.

Terraform apply failed?

If you see a timeout error, your cluster may not have enough memory. Confirm at least 8GB is allocated in Rancher Desktop under Preferences > Virtual Machine.

If you see a conflict error mentioning Traefik or port 80, confirm Traefik is disabled. Run kubectl get pods -n kube-system and check that no Traefik pods are listed.

Help me debug my terraform apply failure.

Verify the platform is running

Now you need to confirm every component started correctly and that you can access the management UIs.

• Check pod status across all platform namespaces by running:

kubectl get pods -n ingress-nginx
kubectl get pods -n argocd
kubectl get pods -n argo-rollouts
kubectl get pods -n monitoring
kubectl get pods -n postgres

All pods should show Running or Completed status. Some pods may take a minute to reach Running if images are still being pulled.

Quick all-namespaces check

You can also run kubectl get pods --all-namespaces to see everything at once. Look for any pods stuck in CrashLoopBackOff or Pending state.

• Retrieve the ArgoCD admin password by running:

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

This prints the auto-generated password. Copy it somewhere safe. The username is admin.

• Access the ArgoCD UI by running:

kubectl port-forward svc/argocd-server -n argocd 8080:443

• Open your browser to https://localhost:8080.
• Accept the self-signed certificate warning.
• Log in with username admin and the password you just retrieved.

You should see the ArgoCD dashboard with an empty application list.

• Open a new terminal tab and access the Grafana UI by running:

kubectl port-forward svc/prometheus-grafana -n monitoring 3000:80

• Open your browser to http://localhost:3000.
• Log in with username admin and password admin.

You should see the Grafana home page with pre-installed Kubernetes dashboards from the kube-prometheus-stack.

• Open another terminal tab and access the Argo Rollouts dashboard by running:

kubectl port-forward svc/argo-rollouts-dashboard -n argo-rollouts 3100:3100

• Open your browser to http://localhost:3100.

You should see the Argo Rollouts dashboard. It will be empty for now since no Rollouts exist yet.

Port-forward immediately exits?

This usually means the target service does not exist yet. Run kubectl get svc -n argocd (or the relevant namespace) to confirm the service is listed. If not, check kubectl get pods -n argocd for pods that have not reached Running status.

Help me figure out why kubectl port-forward exits immediately.

Your entire platform is now running from a single terraform apply command. Next up, you will build the FastAPI inference service that ties into this platform through Prometheus metrics and PostgreSQL logging.

Build the LLM Inference Service

Your platform stack is running. Now you need the application that actually serves predictions.

In this step, you will build a FastAPI inference service that proxies requests to Ollama, logs every prediction to PostgreSQL, and exports custom Prometheus metrics. You will containerize it, test it locally, then push the image to GitHub Container Registry so it is ready for deployment.

In this step, get ready to:

• Create the FastAPI inference service with Prometheus metrics and PostgreSQL prediction logging.
• Build and test the Docker image locally against Ollama and PostgreSQL.
• Push the container image to GitHub Container Registry.

Create the FastAPI inference application

The inference service is the core workload your GitOps pipeline will deploy. It needs three capabilities: proxy inference requests to Ollama, log predictions to PostgreSQL for drift analysis, and export Prometheus metrics so Argo Rollouts can evaluate canary health.

• Create the app/ directory inside your project folder by running:

mkdir -p app

• Create the dependencies file app/requirements.txt with the following content:

fastapi
uvicorn
prometheus-client
psycopg2-binary
httpx

What are these dependencies?

• fastapi is the async web framework for your API endpoints.
• uvicorn is the ASGI server that runs the FastAPI app.
• prometheus-client provides Counter, Histogram, and Gauge metric types that Prometheus scrapes.
• psycopg2-binary is the PostgreSQL adapter for logging predictions.
• httpx is an async HTTP client for proxying requests to Ollama.

• Create app/main.py and add the imports, configuration, and Prometheus metric definitions:

import os
import time
import random
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse, Response
from prometheus_client import Counter, Histogram, Gauge, generate_latest, CONTENT_TYPE_LATEST
import psycopg2
import httpx

# Configuration from environment variables
OLLAMA_HOST = os.getenv("OLLAMA_HOST", "http://localhost:11434")
MODEL_NAME = os.getenv("MODEL_NAME", "qwen2:0.5b")
MODEL_VERSION = os.getenv("MODEL_VERSION", "v1")
CHAOS_MODE = os.getenv("CHAOS_MODE", "false").lower() == "true"
DB_HOST = os.getenv("DB_HOST", "localhost")
DB_PORT = os.getenv("DB_PORT", "5432")
DB_NAME = os.getenv("DB_NAME", "predictions")
DB_USER = os.getenv("DB_USER", "postgres")
DB_PASSWORD = os.getenv("DB_PASSWORD", "postgres")

# Prometheus metrics
request_count = Counter(
    "request_count",
    "Total inference requests",
    ["status", "model_version"]
)
request_latency = Histogram(
    "request_latency",
    "Inference request latency in seconds",
    ["model_version"]
)
model_info = Gauge(
    "model_info",
    "Model version information",
    ["model_version"]
)

# Set model version gauge to 1 so Prometheus knows which version is active
model_info.labels(model_version=MODEL_VERSION).set(1)

app = FastAPI()

What does this code do?

• All configuration is read from environment variables with sensible defaults. This lets you change behavior (Ollama host, model, database credentials) without modifying code.
• request_count is a Counter labeled by status and model_version. Argo Rollouts will query this metric to calculate the success rate during canary analysis.
• request_latency is a Histogram labeled by model_version. This powers the p50/p95/p99 latency panels in Grafana.
• model_info is a Gauge that marks the active model version. Setting it to 1 tells Prometheus which version is serving.
• CHAOS_MODE is read but not activated yet. You will use it in the Secret Mission to deliberately degrade responses.

• Below the app = FastAPI() line, add the database helper functions and startup event:

def get_db_connection():
    """Create a PostgreSQL database connection."""
    return psycopg2.connect(
        host=DB_HOST,
        port=DB_PORT,
        dbname=DB_NAME,
        user=DB_USER,
        password=DB_PASSWORD
    )

def init_db():
    """Create predictions table if it doesn't exist."""
    conn = get_db_connection()
    cur = conn.cursor()
    cur.execute("""
        CREATE TABLE IF NOT EXISTS predictions (
            id SERIAL PRIMARY KEY,
            prompt TEXT NOT NULL,
            response TEXT,
            latency_ms FLOAT,
            model_version VARCHAR(50),
            timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
            status VARCHAR(20)
        )
    """)
    conn.commit()
    cur.close()
    conn.close()

@app.on_event("startup")
async def startup():
    """Initialize database on startup."""
    init_db()

What does this code do?

• get_db_connection() creates a fresh PostgreSQL connection using the environment variables.
• init_db() creates the predictions table if it does not already exist. This table stores every inference request for long-term drift analysis.
• The @app.on_event("startup") decorator ensures the table is created as soon as the service starts, before it receives any requests.

• Below the startup function, add the main /infer endpoint:

@app.post("/infer")
async def infer(request: Request):
    """Handle inference requests by proxying to Ollama."""
    body = await request.json()
    prompt = body.get("prompt", "")
    start_time = time.time()

    try:
        # Proxy request to Ollama
        async with httpx.AsyncClient(timeout=60.0) as client:
            ollama_response = await client.post(
                f"{OLLAMA_HOST}/api/generate",
                json={"model": MODEL_NAME, "prompt": prompt, "stream": False}
            )
            ollama_response.raise_for_status()
            result = ollama_response.json()
            response_text = result.get("response", "")

        latency_ms = (time.time() - start_time) * 1000

        # Log prediction to PostgreSQL
        conn = get_db_connection()
        cur = conn.cursor()
        cur.execute(
            "INSERT INTO predictions (prompt, response, latency_ms, model_version, status) VALUES (%s, %s, %s, %s, %s)",
            (prompt, response_text, latency_ms, MODEL_VERSION, "success")
        )
        conn.commit()
        cur.close()
        conn.close()

        # Record Prometheus metrics
        request_count.labels(status="success", model_version=MODEL_VERSION).inc()
        request_latency.labels(model_version=MODEL_VERSION).observe(latency_ms / 1000)

        return JSONResponse({
            "response": response_text,
            "model_version": MODEL_VERSION,
            "latency_ms": latency_ms
        })

    except Exception as e:
        latency_ms = (time.time() - start_time) * 1000
        request_count.labels(status="error", model_version=MODEL_VERSION).inc()
        request_latency.labels(model_version=MODEL_VERSION).observe(latency_ms / 1000)

        # Log failed prediction
        try:
            conn = get_db_connection()
            cur = conn.cursor()
            cur.execute(
                "INSERT INTO predictions (prompt, response, latency_ms, model_version, status) VALUES (%s, %s, %s, %s, %s)",
                (prompt, str(e), latency_ms, MODEL_VERSION, "error")
            )
            conn.commit()
            cur.close()
            conn.close()
        except Exception:
            pass

        return JSONResponse(
            {"error": str(e), "model_version": MODEL_VERSION},
            status_code=500
        )

What does the /infer endpoint do?

This is the core business logic of your service. It does three things in sequence:

• Proxies the prompt to Ollama's /api/generate endpoint with streaming disabled. The 60-second timeout accounts for slow model inference.
• Logs the prediction (prompt, response, latency, model version, status) to the predictions table for drift analysis.
• Records Prometheus metrics on every request. Both success and error paths update the counter and histogram, ensuring the AnalysisTemplate always has data to evaluate.

• Finally, add the /metrics and /health endpoints below the /infer function:

@app.get("/metrics")
async def metrics():
    """Expose Prometheus metrics."""
    return Response(
        content=generate_latest(),
        media_type=CONTENT_TYPE_LATEST
    )

@app.get("/health")
async def health():
    """Health check endpoint."""
    return {"status": "healthy", "model_version": MODEL_VERSION}

What do these endpoints do?

• /metrics returns all registered Prometheus metrics in the text exposition format. The ServiceMonitor you will create later tells Prometheus to scrape this endpoint.
• /health is a lightweight liveness check that Kubernetes readiness probes will use.

• Save app/main.py.
• Confirm the file structure exists by running:

ls app/

You should see main.py and requirements.txt in the output.

Don't see both files?

• Make sure you created both files inside the app/ directory, not the project root.
• Check for typos in the filename. The requirements file must be named exactly requirements.txt.

Help me figure out why my app directory is missing files.

✔️ Awesome, I've got everything!

Great. Make sure both files are saved before moving on.

ⓧ I'd like to double check the full code

import os
import time
import random
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse, Response
from prometheus_client import Counter, Histogram, Gauge, generate_latest, CONTENT_TYPE_LATEST
import psycopg2
import httpx

# Configuration from environment variables
OLLAMA_HOST = os.getenv("OLLAMA_HOST", "http://localhost:11434")
MODEL_NAME = os.getenv("MODEL_NAME", "qwen2:0.5b")
MODEL_VERSION = os.getenv("MODEL_VERSION", "v1")
CHAOS_MODE = os.getenv("CHAOS_MODE", "false").lower() == "true"
DB_HOST = os.getenv("DB_HOST", "localhost")
DB_PORT = os.getenv("DB_PORT", "5432")
DB_NAME = os.getenv("DB_NAME", "predictions")
DB_USER = os.getenv("DB_USER", "postgres")
DB_PASSWORD = os.getenv("DB_PASSWORD", "postgres")

# Prometheus metrics
request_count = Counter(
    "request_count",
    "Total inference requests",
    ["status", "model_version"]
)
request_latency = Histogram(
    "request_latency",
    "Inference request latency in seconds",
    ["model_version"]
)
model_info = Gauge(
    "model_info",
    "Model version information",
    ["model_version"]
)

# Set model version gauge to 1 so Prometheus knows which version is active
model_info.labels(model_version=MODEL_VERSION).set(1)

app = FastAPI()

def get_db_connection():
    """Create a PostgreSQL database connection."""
    return psycopg2.connect(
        host=DB_HOST,
        port=DB_PORT,
        dbname=DB_NAME,
        user=DB_USER,
        password=DB_PASSWORD
    )

def init_db():
    """Create predictions table if it doesn't exist."""
    conn = get_db_connection()
    cur = conn.cursor()
    cur.execute("""
        CREATE TABLE IF NOT EXISTS predictions (
            id SERIAL PRIMARY KEY,
            prompt TEXT NOT NULL,
            response TEXT,
            latency_ms FLOAT,
            model_version VARCHAR(50),
            timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
            status VARCHAR(20)
        )
    """)
    conn.commit()
    cur.close()
    conn.close()

@app.on_event("startup")
async def startup():
    """Initialize database on startup."""
    init_db()

@app.post("/infer")
async def infer(request: Request):
    """Handle inference requests by proxying to Ollama."""
    body = await request.json()
    prompt = body.get("prompt", "")
    start_time = time.time()

    try:
        # Proxy request to Ollama
        async with httpx.AsyncClient(timeout=60.0) as client:
            ollama_response = await client.post(
                f"{OLLAMA_HOST}/api/generate",
                json={"model": MODEL_NAME, "prompt": prompt, "stream": False}
            )
            ollama_response.raise_for_status()
            result = ollama_response.json()
            response_text = result.get("response", "")

        latency_ms = (time.time() - start_time) * 1000

        # Log prediction to PostgreSQL
        conn = get_db_connection()
        cur = conn.cursor()
        cur.execute(
            "INSERT INTO predictions (prompt, response, latency_ms, model_version, status) VALUES (%s, %s, %s, %s, %s)",
            (prompt, response_text, latency_ms, MODEL_VERSION, "success")
        )
        conn.commit()
        cur.close()
        conn.close()

        # Record Prometheus metrics
        request_count.labels(status="success", model_version=MODEL_VERSION).inc()
        request_latency.labels(model_version=MODEL_VERSION).observe(latency_ms / 1000)

        return JSONResponse({
            "response": response_text,
            "model_version": MODEL_VERSION,
            "latency_ms": latency_ms
        })

    except Exception as e:
        latency_ms = (time.time() - start_time) * 1000
        request_count.labels(status="error", model_version=MODEL_VERSION).inc()
        request_latency.labels(model_version=MODEL_VERSION).observe(latency_ms / 1000)

        # Log failed prediction
        try:
            conn = get_db_connection()
            cur = conn.cursor()
            cur.execute(
                "INSERT INTO predictions (prompt, response, latency_ms, model_version, status) VALUES (%s, %s, %s, %s, %s)",
                (prompt, str(e), latency_ms, MODEL_VERSION, "error")
            )
            conn.commit()
            cur.close()
            conn.close()
        except Exception:
            pass

        return JSONResponse(
            {"error": str(e), "model_version": MODEL_VERSION},
            status_code=500
        )

@app.get("/metrics")
async def metrics():
    """Expose Prometheus metrics."""
    return Response(
        content=generate_latest(),
        media_type=CONTENT_TYPE_LATEST
    )

@app.get("/health")
async def health():
    """Health check endpoint."""
    return {"status": "healthy", "model_version": MODEL_VERSION}

Write the Dockerfile and test locally

Before deploying to Kubernetes, you need to verify the service works end-to-end. Building and running a local container confirms both Ollama connectivity and PostgreSQL writes.

• Create a Dockerfile in your project root (the same level as the app/ directory) with the following content:

FROM python:3.11-slim

# Copy application code
COPY app/ /app

# Install Python dependencies
RUN pip install -r /app/requirements.txt

# Run the FastAPI server
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8080"]

What does this Dockerfile do?

• FROM python:3.11-slim uses a minimal Python 3.11 base image to keep the container small.
• COPY app/ /app puts your application code into the container at /app.
• The CMD starts uvicorn on port 8080, which your Kubernetes service will target.

• Build the container image by running:

docker build -t llm-inference:v1 .

• You should see the build complete with a success message. Confirm the image exists by running:

docker images | grep llm-inference

You should see llm-inference with the v1 tag listed.

Docker build failing?

• Make sure the Dockerfile is in the project root, not inside app/.
• Check that app/requirements.txt exists. The COPY app/ /app line copies the entire directory, so the file must be inside app/.
• If you see a network error during pip install, check your internet connection. Docker needs to download packages during the build.

Help me debug my Docker build failure.

Now test the service end-to-end. You need PostgreSQL accessible from the container, so port-forward the in-cluster PostgreSQL first.

• Open a new terminal and port-forward PostgreSQL by running:

kubectl port-forward -n postgres svc/postgresql 5432:5432

• Leave that running. In your original terminal, start the container with --network host so it can reach both Ollama (on localhost:11434) and the port-forwarded PostgreSQL (on localhost:5432):

docker run --network host \
  -e OLLAMA_HOST=http://localhost:11434 \
  -e DB_HOST=localhost \
  -e DB_PORT=5432 \
  -e DB_NAME=predictions \
  -e DB_USER=postgres \
  -e DB_PASSWORD=postgres \
  -e MODEL_NAME=qwen2:0.5b \
  -e MODEL_VERSION=v1 \
  llm-inference:v1

Why --network host?

On Linux, --network host lets the container share the host's network stack. This means the container can reach Ollama at localhost:11434 and the port-forwarded PostgreSQL at localhost:5432 without extra configuration.

• You should see uvicorn start with output like Uvicorn running on http://0.0.0.0:8080.
• Open another terminal and send a test inference request:

curl -X POST http://localhost:8080/infer \
  -H "Content-Type: application/json" \
  -d '{"prompt": "What is 2+2?"}'

You should receive a JSON response containing a response field with the model's answer, a model_version of v1, and a latency_ms value.

• Verify the prediction was logged to PostgreSQL by running:

kubectl exec -n postgres deploy/postgresql -- psql -U postgres -d predictions -c "SELECT id, prompt, status, model_version FROM predictions LIMIT 5;"

You should see at least one row with your prompt and status success.

• Verify the Prometheus metrics endpoint works:

curl http://localhost:8080/metrics | grep request_count

You should see request_count_total lines showing the counter incremented.

• Stop the running container with Ctrl+C in the terminal where it is running.

Connection refused from the container?

• If Ollama returns connection refused, confirm you configured OLLAMA_HOST=0.0.0.0 via the systemd override in Step 1. Run curl http://localhost:11434 from the host to check Ollama is reachable.
• If PostgreSQL returns connection refused, make sure the port-forward command is still running in the other terminal.
• If you see a database does not exist error, verify the Bitnami PostgreSQL Helm chart was configured with a database named predictions in your Terraform configuration.

Help me debug container connectivity issues with Ollama or PostgreSQL.

Push the image to GitHub Container Registry

GitHub Actions will build and push new images automatically in the next step. For now, you need the initial v1 image in the registry so ArgoCD has something to deploy.

• Log in to GitHub Container Registry using a Personal Access Token (PAT) with the write:packages scope. Generate a PAT from GitHub > Settings > Developer settings > Personal access tokens > Tokens (classic) if you do not have one.
• Run the login command:

docker login ghcr.io

• When prompted, enter your GitHub username and paste your PAT as the password.

Why ghcr.io?

GitHub Container Registry is free for public repositories and integrates natively with GitHub Actions. Your CI workflow will push new images here automatically, and since the repository is public, Kubernetes can pull images without an imagePullSecret.

• Tag the image for your GitHub Container Registry by running:

docker tag llm-inference:v1 ghcr.io/[[GITHUB_USERNAME="your-github-username"]]/llm-inference:v1

• Push the image by running:

docker push ghcr.io/[[GITHUB_USERNAME="your-github-username"]]/llm-inference:v1

You should see each layer pushed and a final digest printed.

• Verify the image is accessible by visiting https://github.com/users/your-github-username/packages/container/package/llm-inference

You should see the v1 tag listed under versions.

Push denied or unauthorized?

• Confirm your PAT has the write:packages scope. Without it, push operations are rejected.
• Make sure the image tag includes your exact GitHub username in lowercase. GitHub Container Registry requires lowercase usernames in the image path.
• If you see a login failure, try running docker login ghcr.io again. Stale credentials can cause authorization errors.

Help me fix my ghcr.io push authorization error.

Your inference service is containerized, tested, and published to a registry. Next up, you will create the GitHub Actions workflow that automates this build-and-push process on every code change.

Set Up GitHub Actions CI Pipeline

In the previous step, you built and tested your inference service container locally, then pushed it to GitHub Container Registry.

Now you need a way to automate that entire build-tag-push cycle so it happens every time you update your code. In this step, you will create a GitHub Actions CI pipeline that builds a new container image on every push to main, tags it with the Git commit SHA, pushes it to ghcr.io, and updates your Kubernetes manifests so ArgoCD can detect the change.

In this step, get ready to:

• Create a GitHub repository with the correct project structure.
• Write a GitHub Actions workflow that builds, tags, and pushes container images automatically.
• Push the initial commit and verify the full CI pipeline runs successfully.

Create the GitHub repository with the project structure

Your CI pipeline needs a specific directory layout. Application code lives at the root, Kubernetes manifests go in a k8s/ directory (which ArgoCD will watch), and the workflow file goes in .github/workflows/.

• Create a new repository on GitHub named llm-inference.
• Set the visibility to Public (this allows pulling container images from ghcr.io without authentication).
• Leave all other defaults and click Create repository.

• In your local project directory, initialize the Git repo and create the required directories by running these commands:

git init
mkdir -p k8s .github/workflows

What does this command do?

The mkdir -p command creates both directories in one go. The -p flag creates parent directories as needed, so .github/workflows/ creates .github/ first, then workflows/ inside it.

• Create the initial Kubernetes Deployment manifest by creating a file at k8s/rollout.yaml with the following content:

# Initial Kubernetes Deployment for the LLM inference service
apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-inference
  labels:
    app: llm-inference
spec:
  replicas: 1
  selector:
    matchLabels:
      app: llm-inference
  template:
    metadata:
      labels:
        app: llm-inference
    spec:
      containers:
        - name: llm-inference
          image: ghcr.io/[[GITHUB_USERNAME="your-github-username"]]/llm-inference:v1
          ports:
            - containerPort: 8080
          env:
            # Ollama host injected via Kubernetes downward API
            - name: OLLAMA_HOST
              valueFrom:
                fieldRef:
                  fieldPath: status.hostIP
            # PostgreSQL connection string
            - name: POSTGRES_URL
              value: "postgresql://llm_user:llm_password@postgresql.default.svc.cluster.local:5432/llm_predictions"
            # Model version label for metrics
            - name: MODEL_VERSION
              value: "v1"

What does this manifest do?

This Deployment tells Kubernetes how to run your inference service. It sets the container image from ghcr.io, exposes port 8080, and injects the environment variables your app needs.

The status.hostIP field uses the Kubernetes downward API to inject the node's IP address. This is how your pod reaches Ollama running on the host machine.

• Confirm the directory structure looks correct by running:

ls -la
ls k8s/
ls .github/workflows/

You should see your app/ directory, Dockerfile, k8s/ directory, and .github/ directory at the root level, with rollout.yaml inside k8s/.

Missing directories?

Make sure you ran mkdir -p with the correct paths. Double-check you are in the same directory that contains your app/ folder and Dockerfile.

Help me debug my project directory structure.

Write the GitHub Actions CI workflow

The CI workflow automates the entire image build and deployment pipeline. It triggers on every push to main, builds a fresh container image, tags it with the exact Git commit SHA for traceability, and pushes it to ghcr.io.

After pushing the image, the workflow updates the image tag in your Kubernetes manifest and commits the change back to the repo. ArgoCD watches this repo, detects the manifest change, and deploys automatically.

• Create the workflow file at .github/workflows/ci.yml with the following content:

name: CI - Build and Push LLM Inference Image

# Trigger on push to main, but skip if commit message contains [skip ci]
on:
  push:
    branches: [main]

jobs:
  build-and-push:
    # Prevent infinite loops from the manifest write-back commit
    if: "!contains(github.event.head_commit.message, '[skip ci]')"
    runs-on: ubuntu-latest

    permissions:
      contents: write   # Needed to push the updated manifest back
      packages: write   # Needed to push container images to ghcr.io

    steps:
      - name: Checkout code
        uses: actions/checkout@v6

      - name: Log in to GitHub Container Registry
        run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin

      - name: Build Docker image
        run: docker build -t ghcr.io/${{ github.repository_owner }}/llm-inference:${{ github.sha }} .

      - name: Push Docker image
        run: docker push ghcr.io/${{ github.repository_owner }}/llm-inference:${{ github.sha }}

      - name: Update image tag in rollout manifest
        run: |
          sed -i "s|image: ghcr.io/.*/llm-inference:.*|image: ghcr.io/${{ github.repository_owner }}/llm-inference:${{ github.sha }}|" k8s/rollout.yaml

      - name: Commit and push updated manifest
        run: |
          git config user.name "github-actions[bot]"
          git config user.email "github-actions[bot]@users.noreply.github.com"
          git add k8s/rollout.yaml
          git diff --cached --quiet || git commit -m "ci: update image tag to ${{ github.sha }} [skip ci]"
          git push

What does this workflow do?

The if condition checks the commit message for [skip ci]. This prevents infinite loops. Without it, the workflow would push an updated manifest, which triggers another push event, which triggers another build, forever.

The permissions block grants contents: write (to commit back to the repo) and packages: write (to push images to ghcr.io). These are the minimum permissions the workflow needs.

The sed command uses a regex to find the current image line in k8s/rollout.yaml and replaces it with the new SHA-tagged image. The pipe character | is used as the delimiter to avoid conflicts with slashes in the image path.

The git diff --cached --quiet || prefix means the commit only happens if there are actual changes. This prevents failures on identical re-runs.

Why tag with the Git SHA?

Tagging images with the commit SHA creates perfect traceability. If something breaks in production, you can look at the running container's tag and know exactly which commit produced it. This is a standard practice in production CI/CD pipelines.

• Save the file at .github/workflows/ci.yml.

YAML indentation errors?

YAML is whitespace-sensitive. Make sure you use spaces (not tabs) and that nested keys are indented by exactly 2 spaces from their parent. The steps array items start with - name: at the same indentation level.

Help me fix YAML indentation in my GitHub Actions workflow.

✔️ Awesome, I've got everything!

Great. Make sure you have saved both k8s/rollout.yaml and .github/workflows/ci.yml.

ⓧ I'd like to double check the full code

Here is the complete .github/workflows/ci.yml:

name: CI - Build and Push LLM Inference Image

# Trigger on push to main, but skip if commit message contains [skip ci]
on:
  push:
    branches: [main]

jobs:
  build-and-push:
    # Prevent infinite loops from the manifest write-back commit
    if: "!contains(github.event.head_commit.message, '[skip ci]')"
    runs-on: ubuntu-latest

    permissions:
      contents: write   # Needed to push the updated manifest back
      packages: write   # Needed to push container images to ghcr.io

    steps:
      - name: Checkout code
        uses: actions/checkout@v6

      - name: Log in to GitHub Container Registry
        run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin

      - name: Build Docker image
        run: docker build -t ghcr.io/${{ github.repository_owner }}/llm-inference:${{ github.sha }} .

      - name: Push Docker image
        run: docker push ghcr.io/${{ github.repository_owner }}/llm-inference:${{ github.sha }}

      - name: Update image tag in rollout manifest
        run: |
          sed -i "s|image: ghcr.io/.*/llm-inference:.*|image: ghcr.io/${{ github.repository_owner }}/llm-inference:${{ github.sha }}|" k8s/rollout.yaml

      - name: Commit and push updated manifest
        run: |
          git config user.name "github-actions[bot]"
          git config user.email "github-actions[bot]@users.noreply.github.com"
          git add k8s/rollout.yaml
          git diff --cached --quiet || git commit -m "ci: update image tag to ${{ github.sha }} [skip ci]"
          git push

And here is the complete k8s/rollout.yaml:

# Initial Kubernetes Deployment for the LLM inference service
apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-inference
  labels:
    app: llm-inference
spec:
  replicas: 1
  selector:
    matchLabels:
      app: llm-inference
  template:
    metadata:
      labels:
        app: llm-inference
    spec:
      containers:
        - name: llm-inference
          image: ghcr.io/[[GITHUB_USERNAME="your-github-username"]]/llm-inference:v1
          ports:
            - containerPort: 8080
          env:
            # Ollama host injected via Kubernetes downward API
            - name: OLLAMA_HOST
              valueFrom:
                fieldRef:
                  fieldPath: status.hostIP
            # PostgreSQL connection string
            - name: POSTGRES_URL
              value: "postgresql://llm_user:llm_password@postgresql.default.svc.cluster.local:5432/llm_predictions"
            # Model version label for metrics
            - name: MODEL_VERSION
              value: "v1"

Push the initial commit and verify the workflow

With the workflow and manifests in place, you can now push everything to GitHub. This initial push triggers the CI workflow for the first time.

• Add the remote origin and push your initial commit by running these commands:

git add .
git commit -m "feat: initial project structure with CI pipeline"
git branch -M main
git remote add origin https://github.com/[[GITHUB_USERNAME="your-github-username"]]/llm-inference.git
git push -u origin main

What happens next?

As soon as you push to main, GitHub detects the workflow file in .github/workflows/ci.yml and runs it. The workflow checks out your code, builds the Docker image, tags it with the commit SHA, pushes it to ghcr.io, then updates k8s/rollout.yaml with the new image tag and commits back.

• Open your repository on GitHub in a browser.
• Click the Actions tab to see the workflow run.
• Click on the running workflow to watch the build progress in real time.

Wait for the workflow to complete. You should see a green checkmark next to each step.

• Verify the container image was pushed by navigating to your repository's Packages section on GitHub (click your profile picture, then Your packages).
• Pull the latest changes (including the workflow's write-back commit) by running:

git pull

• Open k8s/rollout.yaml and verify the image tag has been updated from v1 to a full commit SHA (a 40-character hex string).

What should I see?

After pulling, the image: line in k8s/rollout.yaml should look something like ghcr.io/yourusername/llm-inference:a1b2c3d4e5f6... where the tag is the full 40-character commit SHA.

The commit history should also show a new commit with a message containing [skip ci].

Workflow failed?

Check the workflow run logs on the Actions tab. Common issues include:

• Permission denied on ghcr.io push: make sure the permissions block in your workflow has packages: write.
• Commit push failed: verify contents: write is set in the permissions block.
• Docker build failed: check that your Dockerfile is at the repository root and that app/ and app/requirements.txt exist.

Help me debug my GitHub Actions CI workflow failure.

Your CI pipeline is now live. Every push to main automatically builds a new image, pushes it, and updates your manifest. Next, you will connect ArgoCD to this repository so it deploys changes automatically when the manifest updates.

Deploy with ArgoCD GitOps

Your CI pipeline can build and push new container images automatically. Now you need a way to get those images deployed to your Kubernetes cluster without manual kubectl apply commands.

This is where ArgoCD shines. It watches your Git repository for changes to Kubernetes manifests and automatically syncs them to the cluster. When GitHub Actions updates the image tag in your manifest file, ArgoCD detects the change and deploys the new version.

In this step, get ready to:

• Log into ArgoCD and create a GitOps Application pointing at your repository.
• Deploy the inference service with Services, Ingress, and ServiceMonitor.
• Verify the full data path from request to Ollama to PostgreSQL to Prometheus.

Log into ArgoCD and create the Application

Before ArgoCD can sync manifests from your repository, you need to authenticate with the ArgoCD server and register your repo as an Application source.

• Retrieve the ArgoCD admin password by running this command:

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

What does this command do?

ArgoCD generates a random admin password at install time and stores it in a Kubernetes Secret called argocd-initial-admin-secret. The command extracts the base64-encoded password field and decodes it so you can use it to log in.

• Copy the password output. You will need it in the next command.
• Log into the ArgoCD CLI by running:

argocd login localhost:8080 --username admin --password <paste-your-password-here> --insecure

Why the --insecure flag?

Your ArgoCD server is running locally without TLS certificates. The --insecure flag skips certificate verification so the CLI can connect over plain HTTP. In production you would use proper TLS.

• Port-forward the ArgoCD server if it is not already accessible on localhost:8080:

kubectl port-forward svc/argocd-server -n argocd 8080:443 &

Now create the ArgoCD Application manifest that tells ArgoCD where to find your Kubernetes manifests.

• Create the file k8s/argocd-app.yaml with the following content:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: llm-inference
  namespace: argocd
spec:
  project: default
  source:
    # Your GitHub repository URL
    repoURL: [[REPO_URL="https://github.com/your-username/llm-inference.git"]]
    targetRevision: main
    path: k8s
  destination:
    server: https://kubernetes.default.svc
    namespace: default
  syncPolicy:
    automated:
      selfHeal: true
      prune: true

What does this manifest do?

• source.repoURL points ArgoCD at your GitHub repository.
• source.path: k8s tells ArgoCD to look only in the k8s/ directory for manifests.
• syncPolicy.automated.selfHeal: true means ArgoCD will revert any manual cluster changes back to what Git says.
• syncPolicy.automated.prune: true means ArgoCD will delete resources that no longer exist in Git.

• Apply the ArgoCD Application to your cluster by running:

kubectl apply -f k8s/argocd-app.yaml

• Verify the Application was created by running:

argocd app get llm-inference

You should see the Application listed with a status of Synced or OutOfSync (it will sync once all manifests are in the k8s/ directory).

Seeing a connection refused error?

Make sure the port-forward to ArgoCD is still running. If it dropped, re-run the kubectl port-forward command above.

help me troubleshoot my ArgoCD CLI connection

Deploy v1 of the inference service via ArgoCD

ArgoCD is now watching your k8s/ directory. You need to add the remaining Kubernetes resources so ArgoCD can deploy the full application stack: two Services (stable and canary), an Ingress, and a ServiceMonitor for Prometheus scraping.

• Create the file k8s/services.yaml with the stable and canary Services:

# Stable service - receives production traffic
apiVersion: v1
kind: Service
metadata:
  name: llm-inference-stable
  labels:
    app: llm-inference
spec:
  selector:
    app: llm-inference
  ports:
    - port: 8080
      targetPort: 8080
      name: http
---
# Canary service - receives canary traffic during rollouts
apiVersion: v1
kind: Service
metadata:
  name: llm-inference-canary
  labels:
    app: llm-inference
spec:
  selector:
    app: llm-inference
  ports:
    - port: 8080
      targetPort: 8080
      name: http

Why two Services?

Argo Rollouts needs separate Service objects for stable and canary traffic. During a canary deployment, the controller routes a percentage of traffic to the canary Service while the rest goes to the stable Service. For now both Services select the same pods. Later when you convert to a Rollout, Argo Rollouts manages the selectors automatically.

• Create the file k8s/ingress.yaml to expose the service via NGINX Ingress:

# Ingress routes external traffic to the stable service
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: llm-inference
spec:
  ingressClassName: nginx
  rules:
    - http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: llm-inference-stable
                port:
                  number: 8080

What does ingressClassName: nginx do?

The ingressClassName field tells Kubernetes which Ingress controller should handle this resource. You installed the NGINX Ingress Controller in Step 2, so setting this to nginx routes all traffic through it. This is also the Ingress that Argo Rollouts will use for canary traffic splitting later.

• Create the file k8s/servicemonitor.yaml so Prometheus discovers and scrapes your app's metrics:

# ServiceMonitor tells Prometheus to scrape the inference service
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: llm-inference
  labels:
    release: prometheus
spec:
  selector:
    matchLabels:
      app: llm-inference
  endpoints:
    - port: http
      path: /metrics
      interval: 15s

What does the ServiceMonitor do?

The ServiceMonitor CRD is part of the kube-prometheus-stack you installed with Terraform. It tells Prometheus to automatically scrape the /metrics endpoint on any Service matching app: llm-inference every 15 seconds.

The release: prometheus label is required so the Prometheus operator's serviceMonitorSelector picks up this ServiceMonitor.

• Commit and push all the new manifests to your repository by running:

git add k8s/
git commit -m "Add services, ingress, and servicemonitor"
git push origin main

• Wait 30-60 seconds for ArgoCD to detect the change and sync.
• Check the sync status by running:

argocd app get llm-inference

You should see the status showing Synced and Healthy.

• Verify the pods are running by running:

kubectl get pods -l app=llm-inference

You should see one pod in Running status.

Pod stuck in CrashLoopBackOff?

Check the pod logs for connection errors. The most common issue is Ollama not being reachable from inside the cluster. Verify that you configured Ollama to listen on 0.0.0.0 in Step 1 and that the systemd service restarted successfully.

help me debug why my llm-inference pod is crashing

Verify the full data path

The deployment is running, but you need to confirm the entire pipeline works end-to-end: a request arrives through NGINX Ingress, hits your FastAPI service, proxies to Ollama, logs the prediction to PostgreSQL, and exports metrics to Prometheus.

• Send a test inference request through the NGINX Ingress by running:

curl -X POST http://localhost/infer \
  -H "Content-Type: application/json" \
  -d '{"prompt": "What is GitOps?", "model": "qwen2:0.5b"}'

You should receive a JSON response containing the model's answer and metadata like latency.

Getting connection refused on port 80?

The NGINX Ingress Controller Service of type LoadBalancer binds to port 80 via k3s ServiceLB. Run kubectl get svc -n ingress-nginx and confirm the EXTERNAL-IP shows an address. If it shows <pending>, make sure Traefik is fully disabled and Rancher Desktop was restarted.

help me troubleshoot NGINX Ingress not binding to port 80

• Verify the prediction was logged in PostgreSQL by running:

kubectl exec -it $(kubectl get pod -l app.kubernetes.io/name=postgresql -o jsonpath='{.items[0].metadata.name}') -- psql -U postgres -d inference -c "SELECT id, prompt, model_version, latency_ms, status FROM predictions ORDER BY id DESC LIMIT 5;"

You should see a row with your What is GitOps? prompt, the model version, latency, and a success status.

What is the predictions table?

Your FastAPI app creates this table on startup and logs every inference request. Each row stores the prompt, response text, latency in milliseconds, model version, timestamp, and HTTP status. This data becomes the foundation for long-term model drift analysis in later steps.

• Confirm metrics are flowing to Prometheus by port-forwarding to the Prometheus UI:

kubectl port-forward svc/prometheus-kube-prometheus-prometheus -n monitoring 9090:9090 &

• Open http://localhost:9090 in your browser.
• In the query box, type request_count_total and click **Execute**.

You should see at least one result showing your inference service's request counter with a value of 1 or more.

No metrics showing in Prometheus?

If the query returns empty results, the ServiceMonitor may not be picked up. Check that the release: prometheus label is present on your ServiceMonitor and that the port name http matches your Service definition.

help me debug why Prometheus is not scraping my ServiceMonitor

Your GitOps pipeline is live. Every change committed to the k8s/ directory in your repository automatically deploys to your cluster within seconds. Next up, you will convert this Deployment into an Argo Rollout with automated canary analysis.

Configure Argo Rollouts Canary Strategy

Your v1 inference service is running, verified end-to-end through NGINX Ingress, Ollama, PostgreSQL, and Prometheus. Now it's time to make deployments safe and automated.

In this step, you'll convert your standard Kubernetes Deployment into an Argo Rollout with a canary strategy. This means future updates won't go to 100% of traffic immediately. Instead, Argo Rollouts will route just 10% of traffic to the new version, run a Prometheus analysis to check error rates, and only promote if the metrics look healthy.

In this step, get ready to:

• Convert the Kubernetes Deployment to an Argo Rollout with a canary strategy and NGINX traffic routing.
• Create an AnalysisTemplate that queries Prometheus for success rate metrics.
• Push the manifests to Git, verify ArgoCD syncs, and confirm the Rollout is healthy.

Convert the Deployment to an Argo Rollout

The Argo Rollout CRD replaces the standard Kubernetes Deployment. It uses the same pod template spec, but adds a strategy.canary section that defines how traffic shifts between versions.

• Open k8s/rollout.yaml and replace its entire contents with the following Argo Rollout manifest:

# Argo Rollout resource - replaces the standard Deployment
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: llm-inference
spec:
  replicas: 2
  selector:
    matchLabels:
      app: llm-inference
  template:
    metadata:
      labels:
        app: llm-inference
    spec:
      containers:
        - name: llm-inference
          image: ghcr.io/[[GITHUB_USERNAME="your-github-username"]]/llm-inference:latest
          ports:
            - containerPort: 8080
          env:
            # Node IP injected via Kubernetes downward API
            - name: OLLAMA_HOST
              valueFrom:
                fieldRef:
                  fieldPath: status.hostIP
            # Connection string for the in-cluster PostgreSQL
            - name: POSTGRES_URL
              value: "postgresql://llmuser:llmpass@postgresql.default.svc.cluster.local:5432/llm_predictions"
            # Model version label for metrics
            - name: MODEL_VERSION
              value: "v1"
            # Chaos mode toggle (used in Secret Mission)
            - name: CHAOS_MODE
              value: "false"
  strategy:
    canary:
      # Services for traffic routing
      canaryService: llm-inference-canary
      stableService: llm-inference-stable
      # NGINX Ingress traffic splitting
      trafficRouting:
        nginx:
          stableIngress: llm-inference
      # Inline analysis blocks the rollout until it completes
      steps:
        - setWeight: 10
        - analysis:
            templates:
              - templateName: success-rate
        - setWeight: 50
        - pause:
            duration: 30s
        - setWeight: 100

What does this manifest do?

The strategy.canary section tells Argo Rollouts how to progressively shift traffic during updates.

• canaryService and stableService point to the two Services you already created. Argo Rollouts injects pod-template-hash selectors into these to route traffic to the correct ReplicaSet.
• trafficRouting.nginx.stableIngress references your existing Ingress. Argo Rollouts automatically creates a second "canary" Ingress with nginx.ingress.kubernetes.io/canary: "true" and canary-weight annotations to split traffic at the NGINX layer.
• The steps define the rollout progression: 10% traffic to canary, run Prometheus analysis (about 2 minutes), then 50%, pause 30 seconds, then promote to 100%.

Why NGINX traffic splitting instead of replica ratio?

Without an ingress controller integration, Argo Rollouts can only approximate traffic weights by adjusting the ratio of canary-to-stable pods. With 2 replicas, the minimum canary weight would be 50%.

NGINX Ingress annotations give you precise percentage-based traffic splitting (10%, 50%, etc.) regardless of how many pods are running. This is the production pattern used by teams managing real ML deployments.

Create the AnalysisTemplate for Prometheus metrics

The AnalysisTemplate defines what metrics Argo Rollouts should check during the canary phase and what thresholds constitute a pass or fail.

• Create a new file k8s/analysis-template.yaml and paste the following:

# AnalysisTemplate - defines the Prometheus query and success criteria
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  metrics:
    - name: success-rate
      # Poll Prometheus every 30 seconds
      interval: 30s
      # Run 4 measurements (4 x 30s = ~2 minutes total)
      count: 4
      # Allow up to 3 consecutive failures before aborting
      failureLimit: 3
      # At least 95% of requests must return HTTP 200
      successCondition: "result[0] >= 0.95"
      provider:
        prometheus:
          # In-cluster Prometheus service URL
          address: http://prometheus-kube-prometheus-prometheus.monitoring:9090
          query: |
            sum(rate(request_count{status="200"}[1m])) /
            sum(rate(request_count[1m]))

How does the analysis decide pass or fail?

• Every 30 seconds, Argo Rollouts runs the PromQL query against Prometheus. The query calculates the ratio of successful requests (status 200) to total requests over a 1-minute window.
• successCondition: "result[0] >= 0.95" means the success rate must be at least 95%. Prometheus returns results as a vector, so result[0] accesses the first (and only) value.
• failureLimit: 3 means the analysis tolerates up to 3 failed measurements before aborting the entire rollout and reverting traffic.
• count: 4 runs 4 total measurements. If all 4 pass (or fewer than 3 fail), the analysis succeeds and the rollout continues to the next step.

• Save both k8s/rollout.yaml and k8s/analysis-template.yaml.

Push to Git and verify the Rollout is healthy

ArgoCD is already watching your k8s/ directory with automated sync enabled. When you push these manifest changes, ArgoCD will detect the new Rollout resource and sync it to the cluster automatically.

• Commit and push the updated manifests by running these commands:

git add k8s/rollout.yaml k8s/analysis-template.yaml
git commit -m "Convert Deployment to Argo Rollout with canary strategy"
git push origin main

• Wait about 30 seconds for ArgoCD to sync, then check the Rollout status by running:

kubectl argo rollouts get rollout llm-inference --watch

What should I see?

The output shows the Rollout name, strategy (Canary), and status. Since this is the initial creation (no upgrade has occurred), the Rollout immediately marks the current version as stable with status "Healthy". You should see all replicas listed under the stable revision with a green checkmark.

Press Ctrl+C to exit the watch when you see the Healthy status.

• Explore the Rollout visually by port-forwarding the Argo Rollouts dashboard:

kubectl argo rollouts dashboard

• Open http://localhost:3100/rollouts in your browser.
• Click on the llm-inference rollout to see the canary steps, current revision, and stable status.

Rollout not showing as Healthy?

• Run argocd app get llm-inference to check if ArgoCD has synced. If the sync status shows "OutOfSync", wait another 30 seconds or trigger a manual sync with argocd app sync llm-inference.
• If you see CRD-related errors, verify Argo Rollouts is running with kubectl get pods -n argo-rollouts. The controller pod must be in Running state to process Rollout resources.
• Check that the Services llm-inference-stable and llm-inference-canary exist with kubectl get svc. The Rollout requires both to be present.

Still stuck? Help me debug why my Argo Rollout isn't showing as Healthy after syncing

✔️ Awesome, I've got everything!

Great. Double check you've saved both files and your Rollout shows Healthy.

ⓧ I'd like to double check the full code

Here is the complete k8s/rollout.yaml:

# Argo Rollout resource - replaces the standard Deployment
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: llm-inference
spec:
  replicas: 2
  selector:
    matchLabels:
      app: llm-inference
  template:
    metadata:
      labels:
        app: llm-inference
    spec:
      containers:
        - name: llm-inference
          image: ghcr.io/[[GITHUB_USERNAME="your-github-username"]]/llm-inference:latest
          ports:
            - containerPort: 8080
          env:
            # Node IP injected via Kubernetes downward API
            - name: OLLAMA_HOST
              valueFrom:
                fieldRef:
                  fieldPath: status.hostIP
            # Connection string for the in-cluster PostgreSQL
            - name: POSTGRES_URL
              value: "postgresql://llmuser:llmpass@postgresql.default.svc.cluster.local:5432/llm_predictions"
            # Model version label for metrics
            - name: MODEL_VERSION
              value: "v1"
            # Chaos mode toggle (used in Secret Mission)
            - name: CHAOS_MODE
              value: "false"
  strategy:
    canary:
      # Services for traffic routing
      canaryService: llm-inference-canary
      stableService: llm-inference-stable
      # NGINX Ingress traffic splitting
      trafficRouting:
        nginx:
          stableIngress: llm-inference
      # Inline analysis blocks the rollout until it completes
      steps:
        - setWeight: 10
        - analysis:
            templates:
              - templateName: success-rate
        - setWeight: 50
        - pause:
            duration: 30s
        - setWeight: 100

Here is the complete k8s/analysis-template.yaml:

# AnalysisTemplate - defines the Prometheus query and success criteria
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  metrics:
    - name: success-rate
      # Poll Prometheus every 30 seconds
      interval: 30s
      # Run 4 measurements (4 x 30s = ~2 minutes total)
      count: 4
      # Allow up to 3 consecutive failures before aborting
      failureLimit: 3
      # At least 95% of requests must return HTTP 200
      successCondition: "result[0] >= 0.95"
      provider:
        prometheus:
          # In-cluster Prometheus service URL
          address: http://prometheus-kube-prometheus-prometheus.monitoring:9090
          query: |
            sum(rate(request_count{status="200"}[1m])) /
            sum(rate(request_count[1m]))

Your automated canary pipeline is armed. Next up, you'll trigger an actual canary release by pushing a v2 update and watching the full pipeline fire: CI builds the image, ArgoCD syncs, and Argo Rollouts begins the progressive rollout with live Prometheus analysis.

Run Load Tests and Trigger a Canary Release

Your canary strategy is deployed and healthy, but it has never been tested with real traffic. Argo Rollouts needs actual metric data flowing through Prometheus to evaluate whether a new version is safe to promote.

In this step, you will generate sustained load against the inference service, push a v2 model update, and watch the entire pipeline fire end-to-end. The AnalysisRun will sample your success rate every 30 seconds and automatically promote through 10%, 50%, and 100% traffic.

In this step, get ready to:

• Create a load generation script that sends sustained traffic during the canary.
• Push a v2 model update and observe the full GitOps pipeline.
• Watch Argo Rollouts automatically promote the canary based on Prometheus metrics.

Create the load generation script

Without traffic, the Prometheus AnalysisRun returns empty results and may mark the canary as Inconclusive. You need a steady stream of requests hitting the inference endpoint so the success rate query has enough data points to evaluate.

• Create the scripts/ directory in your project root by running:

mkdir -p scripts

• Create the load test script by running:

cat > scripts/load-test.sh << 'EOF'
#!/bin/bash
# Sends sustained POST requests to the inference endpoint
# to ensure Prometheus has data during the canary analysis

INGRESS_IP=$(kubectl get svc -n ingress-nginx ingress-nginx-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

echo "Starting load test against $INGRESS_IP..."
echo "Press Ctrl+C to stop."

hey -z 10m -c 1 -q 2 -m POST \
  -T "application/json" \
  -d '{"prompt": "Explain what a canary deployment is in one sentence."}' \
  -t 60 \
  http://$INGRESS_IP/infer
EOF

What does this script do?

The script retrieves the NGINX Ingress external IP from your cluster, then uses hey to send POST requests for 10 minutes.

• -z 10m runs the load test for 10 minutes.
• -c 1 -q 2 uses 1 worker sending 2 requests per second. This produces approximately 2 total requests per second.
• -t 60 sets a 60-second timeout per request, since LLM inference can be slow.

• Make the script executable by running:

chmod +x scripts/load-test.sh

• Confirm the file exists by running:

ls scripts/

You should see load-test.sh listed in the output.

Script not running?

If you get a permission denied error, confirm you ran chmod +x scripts/load-test.sh. If hey is not found, revisit Step 1 where you installed it.

Help me debug my load test script

Push a v2 model update and start the load generator

To trigger the full pipeline, you need a code change that produces a new container image. Changing the system prompt in your inference service counts as a "model update" because it alters how the LLM responds to queries.

• In app/main.py, find the system prompt string in your Ollama proxy call.
• Update it to a new v2 prompt. For example, change it to instruct the model to respond more concisely or in a different style.

Why change the system prompt?

In production MLOps, a "model update" can mean a new model version, new fine-tuning weights, or new prompt engineering. Changing the system prompt is the simplest way to produce a meaningfully different v2 that the canary pipeline can evaluate.

• Start the load generator in a dedicated terminal by running:

./scripts/load-test.sh

Leave this terminal running. It will continue sending requests throughout the canary release.

• In a separate terminal, commit and push the v2 change by running:

git add app/main.py
git commit -m "feat: update system prompt to v2"
git push origin main

What happens after you push?

The push triggers your GitHub Actions workflow. It builds a new Docker image tagged with the commit SHA, pushes it to ghcr.io, updates the image tag in k8s/rollout.yaml, and commits back with [skip ci].

ArgoCD detects the manifest change and syncs it to your cluster. Argo Rollouts then begins the canary strategy, routing 10% of traffic to the new v2 pods.

• Wait 2-3 minutes for GitHub Actions to complete the build.
• Verify the workflow succeeded in your GitHub repository under the **Actions** tab.

GitHub Actions workflow not triggering?

Confirm your commit message does not contain [skip ci]. Only the automated tag-update commit should include that string. Also verify your workflow file is at .github/workflows/ci.yml and triggers on push to main.

Help me debug my CI pipeline

Observe the canary promotion

Once ArgoCD syncs the updated manifest, Argo Rollouts starts the canary progression. You can watch it in real time from the CLI and the dashboard simultaneously.

• Watch the rollout status in real time by running:

kubectl argo rollouts get rollout llm-inference --watch

What should you see during the canary?

The output shows each step of your canary strategy in real time:

• Status changes to "Progressing" with 10% weight on the canary.
• An AnalysisRun starts, sampling Prometheus every 30 seconds for your success rate metric.
• After 2 minutes of passing analysis, weight increases to 50%.
• After 30 more seconds, weight reaches 100% and the rollout is fully promoted.

• In another terminal, open the Argo Rollouts dashboard to see the visual progression by running:

kubectl argo rollouts dashboard

• Open http://localhost:3100/rollout/default/llm-inference in your browser.
• You should see the rollout progressing through its steps with a live traffic split indicator.

• Verify the AnalysisRun completed successfully by running:

kubectl get analysisrun -l rollouts-pod-template-hash

You should see at least one AnalysisRun with status Successful.

• Confirm the rollout is fully promoted by running:

kubectl argo rollouts get rollout llm-inference

The status should show "Healthy" with all replicas on the new revision. The canary weight should be at 100%.

Rollout stuck at 10% or showing Inconclusive?

If the AnalysisRun shows "Inconclusive", your Prometheus query is returning empty data. Confirm your load generator is still running and sending traffic. Also verify the ServiceMonitor is scraping metrics by checking http://localhost:9090/targets (port-forward Prometheus if needed).

If the rollout shows "Degraded", your success rate dropped below 95%. Check the inference service logs with kubectl logs to see if Ollama is responding correctly.

Help me debug my canary rollout

You have now witnessed the complete GitOps MLOps loop in action. Next up, you will build a Grafana dashboard with alerting rules to visualize the metrics that drove this canary decision.

Build the Grafana Dashboard and Alerting

Your canary pipeline is now proven end-to-end. You watched traffic shift from 10% to 100% as Prometheus confirmed the success rate stayed healthy.

The next piece of the puzzle is visibility. Right now, the metrics exist in Prometheus but you have no persistent way to visualize trends, compare model versions, or get alerted when something degrades. In this step, you will build a Grafana dashboard that surfaces latency, error rate, and model drift, then wire up a PrometheusRule that fires an alert if error rate crosses a threshold.

In this step, get ready to:

• Create a Grafana dashboard ConfigMap with latency, error rate, throughput, and model drift panels.
• Deploy a PrometheusRule CRD that fires an alert when error rate exceeds 5% for more than 2 minutes.
• Query the PostgreSQL prediction log for long-term drift analysis between model versions.

Create the Grafana dashboard ConfigMap

Grafana's sidecar container watches for ConfigMaps with the label grafana_dashboard: "1" and automatically loads any JSON dashboard it finds. This is how kube-prometheus-stack provisions its built-in dashboards.

• Create a new file called k8s/grafana-dashboard-configmap.yaml by running:

touch k8s/grafana-dashboard-configmap.yaml

• Open k8s/grafana-dashboard-configmap.yaml and paste the following ConfigMap with an embedded Grafana dashboard JSON:

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-dashboard-llm-inference
  namespace: monitoring
  labels:
    grafana_dashboard: "1"
data:
  llm-inference.json: |-
    {
      "annotations": { "list": [] },
      "editable": true,
      "fiscalYearStartMonth": 0,
      "graphTooltip": 1,
      "links": [],
      "panels": [
        {
          "title": "Inference Latency by Model Version (p50/p95/p99)",
          "type": "timeseries",
          "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 },
          "datasource": { "type": "prometheus", "uid": "prometheus" },
          "targets": [
            {
              "expr": "histogram_quantile(0.50, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, model_version))",
              "legendFormat": "p50 - {{model_version}}"
            },
            {
              "expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, model_version))",
              "legendFormat": "p95 - {{model_version}}"
            },
            {
              "expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, model_version))",
              "legendFormat": "p99 - {{model_version}}"
            }
          ],
          "fieldConfig": { "defaults": { "unit": "s" } }
        },
        {
          "title": "Error Rate by Model Version",
          "type": "timeseries",
          "gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 },
          "datasource": { "type": "prometheus", "uid": "prometheus" },
          "targets": [
            {
              "expr": "sum(rate(http_requests_total{status=~\"5..\"}[5m])) by (model_version) / sum(rate(http_requests_total[5m])) by (model_version)",
              "legendFormat": "error rate - {{model_version}}"
            }
          ],
          "fieldConfig": { "defaults": { "unit": "percentunit" } }
        },
        {
          "title": "Request Throughput",
          "type": "timeseries",
          "gridPos": { "h": 8, "w": 12, "x": 0, "y": 8 },
          "datasource": { "type": "prometheus", "uid": "prometheus" },
          "targets": [
            {
              "expr": "sum(rate(http_requests_total[5m])) by (model_version)",
              "legendFormat": "{{model_version}}"
            }
          ],
          "fieldConfig": { "defaults": { "unit": "reqps" } }
        },
        {
          "title": "Model Drift: Latency Delta (Canary - Stable)",
          "type": "timeseries",
          "gridPos": { "h": 8, "w": 12, "x": 12, "y": 8 },
          "datasource": { "type": "prometheus", "uid": "prometheus" },
          "targets": [
            {
              "expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{model_version=\"v2\"}[5m])) by (le)) - histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{model_version=\"v1\"}[5m])) by (le))",
              "legendFormat": "p95 latency delta (v2 - v1)"
            }
          ],
          "fieldConfig": { "defaults": { "unit": "s" } }
        }
      ],
      "schemaVersion": 39,
      "title": "LLM Inference Service",
      "uid": "llm-inference-dashboard",
      "version": 1
    }

What does this ConfigMap do?

The grafana_dashboard: "1" label tells the Grafana sidecar (deployed by kube-prometheus-stack) to pick up this ConfigMap and load the JSON as a dashboard.

The four panels cover latency percentiles split by model_version, per-version error rate, total throughput, and a drift comparison panel that subtracts stable latency from canary latency. A positive delta means the canary is slower.

• Apply the ConfigMap to your cluster by running:

kubectl apply -f k8s/grafana-dashboard-configmap.yaml

• Port-forward to Grafana to verify the dashboard loaded:

kubectl port-forward -n monitoring svc/monitoring-grafana 3000:80

• Open http://localhost:3000 in your browser. Log in with username admin and password prom-operator.
• Navigate to **Dashboards** in the left sidebar. You should see LLM Inference Service listed.
• Click into it. All four panels should display data from your recent canary rollout.

Dashboard not appearing in Grafana?

Verify the ConfigMap exists in the monitoring namespace with the correct label by running kubectl get configmap -n monitoring -l grafana_dashboard=1. Your ConfigMap should appear in the list.

If it exists but Grafana still does not show it, restart the Grafana pod to force the sidecar to rescan: kubectl rollout restart deployment -n monitoring monitoring-grafana.

Why isn't my Grafana dashboard ConfigMap being loaded by the sidecar?

Create the PrometheusRule for error rate alerting

A PrometheusRule CRD lets you define alerting rules that the Prometheus Operator automatically loads into Prometheus. When the PromQL expression evaluates to true for the specified duration, Prometheus fires the alert to Alertmanager.

• Create a new file called k8s/prometheus-rule.yaml by running:

touch k8s/prometheus-rule.yaml

• Open k8s/prometheus-rule.yaml and paste the following alerting rule:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: llm-inference-alerts
  namespace: default
  labels:
    # Label must match the Prometheus Operator's ruleSelector
    release: monitoring
spec:
  groups:
    - name: llm-inference
      rules:
        - alert: HighErrorRate
          # Fires when error rate exceeds 5%
          expr: |
            sum(rate(http_requests_total{status=~"5.."}[5m]))
            /
            sum(rate(http_requests_total[5m]))
            > 0.05
          for: 2m
          labels:
            severity: warning
          annotations:
            summary: "High error rate on LLM inference service"
            description: "Error rate has exceeded 5% for more than 2 minutes."

What does this rule do?

The PromQL expression divides 5xx responses by total requests over a 5-minute window. If that ratio exceeds 0.05 (5%) for a continuous 2-minute period, Prometheus fires the HighErrorRate alert with severity warning.

The release: monitoring label ensures the Prometheus Operator (installed via kube-prometheus-stack with that release name) picks up this rule. Without the matching label, the rule is ignored.

• Apply the PrometheusRule to your cluster by running:

kubectl apply -f k8s/prometheus-rule.yaml

• Verify Prometheus loaded the rule by running:

kubectl get prometheusrule llm-inference-alerts

• You should see the rule listed with its name and age. You can also confirm it in the Grafana Alerting panel by navigating to **Alerting** in the left sidebar.

PrometheusRule not picked up by Prometheus?

Check that the release: monitoring label on the PrometheusRule matches the label selector your Prometheus Operator is configured to watch. Run kubectl get prometheus -A -o yaml and look at the ruleSelector field to see which labels it expects.

Help me figure out why my PrometheusRule is not being loaded by the Prometheus Operator.

Query PostgreSQL for long-term drift analysis

Your inference service has been logging every prediction to PostgreSQL since Step 5. This prediction log is your source of truth for long-term model drift analysis that goes beyond what Prometheus retains.

• Connect to the PostgreSQL pod by running:

kubectl exec -it $(kubectl get pod -l app.kubernetes.io/name=postgresql -o jsonpath='{.items[0].metadata.name}') -- psql -U postgres -d llm_inference

• Run a query to compare average response times between v1 and v2:

-- Compare average latency and token count between model versions
SELECT
  model_version,
  COUNT(*) AS total_predictions,
  ROUND(AVG(latency_ms)::numeric, 2) AS avg_latency_ms,
  ROUND(AVG(response_tokens)::numeric, 2) AS avg_tokens
FROM predictions
GROUP BY model_version
ORDER BY model_version;

What does this query show?

This aggregates all logged predictions by model_version, giving you the total count, average latency, and average token count for each version. A significant latency increase or token count change between versions indicates model drift that warrants investigation.

Unlike Prometheus (which typically retains 15 days of metrics), this PostgreSQL log retains the full prediction history. You can query it over weeks or months to detect gradual degradation patterns.

• You should see a table with rows for v1 and v2, each showing prediction count, average latency, and average response tokens.
• Exit the PostgreSQL shell by typing:

\q

Extend this for production

In a real deployment, you would schedule this query (or a variant with time-window filters) as a daily cron job. Export the results to a dashboard or alerting system to catch slow drift that Prometheus real-time metrics might miss.

✔️ Awesome, I've got everything!

Great. Make sure you have saved both k8s/grafana-dashboard-configmap.yaml and k8s/prometheus-rule.yaml.

ⓧ I'd like to double check the full code

Here is the complete k8s/grafana-dashboard-configmap.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-dashboard-llm-inference
  namespace: monitoring
  labels:
    grafana_dashboard: "1"
data:
  llm-inference.json: |-
    {
      "annotations": { "list": [] },
      "editable": true,
      "fiscalYearStartMonth": 0,
      "graphTooltip": 1,
      "links": [],
      "panels": [
        {
          "title": "Inference Latency by Model Version (p50/p95/p99)",
          "type": "timeseries",
          "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 },
          "datasource": { "type": "prometheus", "uid": "prometheus" },
          "targets": [
            {
              "expr": "histogram_quantile(0.50, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, model_version))",
              "legendFormat": "p50 - {{model_version}}"
            },
            {
              "expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, model_version))",
              "legendFormat": "p95 - {{model_version}}"
            },
            {
              "expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, model_version))",
              "legendFormat": "p99 - {{model_version}}"
            }
          ],
          "fieldConfig": { "defaults": { "unit": "s" } }
        },
        {
          "title": "Error Rate by Model Version",
          "type": "timeseries",
          "gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 },
          "datasource": { "type": "prometheus", "uid": "prometheus" },
          "targets": [
            {
              "expr": "sum(rate(http_requests_total{status=~\"5..\"}[5m])) by (model_version) / sum(rate(http_requests_total[5m])) by (model_version)",
              "legendFormat": "error rate - {{model_version}}"
            }
          ],
          "fieldConfig": { "defaults": { "unit": "percentunit" } }
        },
        {
          "title": "Request Throughput",
          "type": "timeseries",
          "gridPos": { "h": 8, "w": 12, "x": 0, "y": 8 },
          "datasource": { "type": "prometheus", "uid": "prometheus" },
          "targets": [
            {
              "expr": "sum(rate(http_requests_total[5m])) by (model_version)",
              "legendFormat": "{{model_version}}"
            }
          ],
          "fieldConfig": { "defaults": { "unit": "reqps" } }
        },
        {
          "title": "Model Drift: Latency Delta (Canary - Stable)",
          "type": "timeseries",
          "gridPos": { "h": 8, "w": 12, "x": 12, "y": 8 },
          "datasource": { "type": "prometheus", "uid": "prometheus" },
          "targets": [
            {
              "expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{model_version=\"v2\"}[5m])) by (le)) - histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{model_version=\"v1\"}[5m])) by (le))",
              "legendFormat": "p95 latency delta (v2 - v1)"
            }
          ],
          "fieldConfig": { "defaults": { "unit": "s" } }
        }
      ],
      "schemaVersion": 39,
      "title": "LLM Inference Service",
      "uid": "llm-inference-dashboard",
      "version": 1
    }

And here is the complete k8s/prometheus-rule.yaml:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: llm-inference-alerts
  namespace: default
  labels:
    # Label must match the Prometheus Operator's ruleSelector
    release: monitoring
spec:
  groups:
    - name: llm-inference
      rules:
        - alert: HighErrorRate
          # Fires when error rate exceeds 5%
          expr: |
            sum(rate(http_requests_total{status=~"5.."}[5m]))
            /
            sum(rate(http_requests_total[5m]))
            > 0.05
          for: 2m
          labels:
            severity: warning
          annotations:
            summary: "High error rate on LLM inference service"
            description: "Error rate has exceeded 5% for more than 2 minutes."

You now have full observability across your inference service. The Grafana dashboard gives you real-time visibility into model performance, the PrometheusRule simulates production alerting, and the PostgreSQL prediction log provides the long-term data backbone for drift detection. Next up, you will explore how this local architecture maps to a production cloud deployment.

Map to Production Architecture

Your local GitOps MLOps pipeline is fully operational. You have a complete feedback loop from code push through CI, container build, GitOps sync, canary release, and Prometheus-driven analysis.

The most powerful aspect of this architecture is portability. Every pattern you built locally transfers directly to a production EKS cluster. In this step, you will review how the same Terraform modules target AWS, understand what changes in production, and confirm the value of what you have built.

In this step, get ready to:

• Review how the local architecture maps to a production EKS deployment.
• Walk through a reference Terraform configuration targeting AWS.
• Understand what stays identical and what changes in production.

Add a reference EKS module to your Terraform configuration

Your existing terraform/main.tf installs Helm releases into your local k3s cluster. In production, the same Terraform workflow targets AWS instead. The terraform-aws-modules/eks/aws module provisions a VPC, EKS cluster, and managed node groups in one declaration.

• Open terraform/providers.tf and add the AWS provider reference at the bottom of the file:

# =============================================================
# PRODUCTION REFERENCE: AWS Provider
# =============================================================
# Uncomment when targeting a real AWS account.

# provider "aws" {
#   region = var.aws_region
# }

What does this do?

This commented block shows where the AWS provider would go when you switch from local Helm releases to cloud infrastructure. The helm and kubernetes providers you already have would then point at the EKS cluster endpoint instead of your local kubeconfig.

• Open terraform/main.tf and add the following reference block at the bottom of the file, below your existing local Helm releases:

# =============================================================
# PRODUCTION REFERENCE: AWS EKS Deployment
# =============================================================
# In production, the Helm releases above deploy identically
# to an EKS cluster. Only the infrastructure target changes.
# Uncomment and configure to deploy to AWS.

# module "vpc" {
#   source  = "terraform-aws-modules/vpc/aws"
#   version = "6.6.1"
#
#   name = "mlops-production-vpc"
#   cidr = "10.0.0.0/16"
#
#   azs             = ["us-east-1a", "us-east-1b", "us-east-1c"]
#   private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
#   public_subnets  = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]
#
#   enable_nat_gateway = true
#   single_nat_gateway = true
# }

# module "eks" {
#   source  = "terraform-aws-modules/eks/aws"
#   version = "21.20.0"
#
#   cluster_name    = "mlops-production"
#   cluster_version = "1.33"
#
#   vpc_id     = module.vpc.vpc_id
#   subnet_ids = module.vpc.private_subnets
#
#   eks_managed_node_groups = {
#     default = {
#       instance_types = ["m5.xlarge"]
#       min_size       = 2
#       max_size       = 5
#       desired_size   = 3
#     }
#   }
# }

What does this reference block show?

The vpc module creates a production VPC with public and private subnets across three availability zones. The NAT gateway allows private-subnet pods to reach the internet for pulling container images.

The eks module provisions a managed Kubernetes cluster at version 1.33 with auto-scaling node groups. Once this is live, your existing ArgoCD and Argo Rollouts Helm releases install identically into EKS. No changes to your k8s/ manifests are needed.

• Save terraform/providers.tf and terraform/main.tf.

Why not run terraform apply on this?

This reference block is commented out intentionally. Running it would provision real AWS resources that cost money. The purpose is to document exactly what changes when you go from local k3s to production EKS. The workflow, manifests, and GitOps patterns remain the same.

Understand what changes in production

Moving from your local cluster to production EKS introduces a small set of infrastructure-level changes. Your application code, Kubernetes manifests, ArgoCD Application, Argo Rollouts canary strategy, and Prometheus analysis all stay identical.

• GitHub Actions pushes to ECR (Elastic Container Registry) instead of ghcr.io. The workflow just swaps the registry URL and adds AWS credential steps.
• DNS uses Route 53 for real domain names instead of localhost port-forwards.
• TLS certificates are managed by cert-manager with Let's Encrypt, replacing the plain HTTP you used locally.
• A Horizontal Pod Autoscaler (HPA) scales pods based on CPU or custom metrics, replacing fixed replica counts.
• Secrets move to AWS Secrets Manager with the External Secrets Operator syncing them into Kubernetes secrets, replacing hardcoded environment variables.
• PostgreSQL moves to RDS (a managed database service), replacing the in-cluster Bitnami chart. This gives you automated backups, replication, and maintenance windows.

What stays exactly the same?

Your k8s/rollout.yaml canary strategy, k8s/analysis-template.yaml Prometheus query, k8s/argocd-app.yaml sync configuration, ServiceMonitor, PrometheusRule alerts, and Grafana dashboard all deploy to EKS without modification. ArgoCD watches the same Git repo. Argo Rollouts performs the same canary with the same traffic splits.

The GitOps principle guarantees this. Your cluster state is defined in Git, not in imperative scripts tied to a specific cluster.

Understand the value of what you built

You now have end-to-end proof that you can design and operate a production-grade ML deployment system. Every concept transfers directly to real EKS or GKE clusters.

• The GitOps workflow (code push triggers sync) is identical in production. Only the Terraform target changes from local Helm to AWS modules.
• The canary strategy (10% to 50% to 100% with Prometheus analysis) works the same way regardless of cluster size or traffic volume.
• The Prometheus metrics, AnalysisTemplate, and automated rollback pattern operate identically in any Kubernetes cluster with kube-prometheus-stack installed.
• The prediction logging and drift analysis pattern scales by pointing at RDS instead of in-cluster PostgreSQL.

How production teams use this pattern

ML platform teams at scale run this exact loop daily. A data scientist pushes a model update. CI builds the container. ArgoCD syncs. Argo Rollouts canary-tests with real production traffic. If the model degrades (latency spike, error rate increase, quality drop), the system rolls back automatically before users notice.

You proved you can build this entire system from scratch. The infrastructure layer (local vs cloud) is the easy part to swap. The hard part is the workflow design, metric-driven automation, and safety guarantees you already have working.

Your local GitOps MLOps pipeline is complete. You have proven every pattern from CI through canary release to automated rollback. Next up, put your safety guarantees to the ultimate test with a chaos engineering challenge.

Secret mission

Chaos Mode: Force an Automated Rollback

Your canary pipeline promoted v2 successfully because it was healthy. But what happens when a bad model update slips through? In this secret mission, you'll deliberately inject failures into your inference service and watch the entire automated safety net activate in real time.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely on your local machine via Rancher Desktop, with free GitHub Actions for CI. There are no ongoing cloud costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

• Rancher Desktop k3s cluster with Helm releases (ArgoCD, Argo Rollouts, kube-prometheus-stack, NGINX Ingress Controller, PostgreSQL).
• Ollama systemd override file at /etc/systemd/system/ollama.service.d/override.conf binding Ollama to 0.0.0.0.
• Local project files (Terraform configs, app code, Kubernetes manifests, scripts).
• GitHub repository with Actions workflow (free, no ongoing cost).
• Container images on ghcr.io (free for public repos).

Keep everything running

No action needed. Choose this if you are still actively building or experimenting with the pipeline.

• Rancher Desktop uses local resources only. Leave it running for further experimentation.
• Your GitHub repository and ghcr.io images are free for public repos and incur no charges.
• The Ollama systemd override remains active, binding Ollama to all interfaces. This does not consume additional resources beyond what Ollama normally uses.
• The Argo Rollouts kubectl plugin and hey binary remain on your system.

Pause - I'll come back to this later

Shut down running processes to free up memory while keeping your files and configuration intact.

• Shut down Rancher Desktop to free the 8GB of RAM allocated to the cluster. Open Rancher Desktop and select File > Exit, or run:

rdctl shutdown

• Revert the Ollama binding so it only listens on localhost while you are not using the cluster:

sudo rm /etc/systemd/system/ollama.service.d/override.conf
sudo systemctl daemon-reload
sudo systemctl restart ollama

• Your Terraform state, project files, GitHub repo, and ghcr.io images remain untouched. When you return, start Rancher Desktop, re-apply the Ollama override, and run terraform apply to bring the platform back up.

Delete - I don't want to use this again

Remove all project resources and start fresh. Follow each item below to fully clean up.

• Destroy all Helm releases and PVCs by running terraform destroy from your Terraform directory:

cd ~/Desktop/gitops-mlops-pipeline/terraform
terraform destroy

• Type yes when prompted to confirm the destruction.
• Factory-reset Rancher Desktop to remove the k3s cluster and all associated data:

rdctl factory-reset

• Remove the Ollama systemd override:

sudo rm /etc/systemd/system/ollama.service.d/override.conf
sudo systemctl daemon-reload
sudo systemctl restart ollama

• Remove the Argo Rollouts kubectl plugin:

sudo rm /usr/local/bin/kubectl-argo-rollouts

• Remove the hey load testing binary (if installed to /usr/local/bin):

sudo rm /usr/local/bin/hey

• Delete your local project files:

rm -rf ~/Desktop/gitops-mlops-pipeline

• Delete the GitHub repository. Go to your repository page on github.com, click Settings, scroll to the bottom under Danger Zone, and click Delete this repository.
• Delete container images from ghcr.io. Go to your GitHub profile, click the Packages tab, find the llm-inference package, click Package settings, then under Danger Zone click Delete this package.

That's a Wrap!

That's a Wrap!

You just built a production-grade MLOps pipeline from scratch. A single git push now cascades through CI, container build, GitOps sync, and a metrics-driven canary release with automated safety guarantees.

You've learned how to:

• Bootstrap an entire platform stack with Terraform IaC, provisioning six Helm charts (ArgoCD, Argo Rollouts, Prometheus, Grafana, NGINX Ingress, PostgreSQL) in one command.
• Build a full GitOps pipeline where code changes automatically flow through GitHub Actions CI, push a container image to ghcr.io, and sync to your Kubernetes cluster via ArgoCD.
• Configure Argo Rollouts canary deployments with real NGINX traffic splitting and Prometheus-driven automated analysis that promotes or rolls back based on live metrics.
• Secret Mission: Inject chaos into a canary release to deliberately trigger and observe automated rollback with full observability across Grafana dashboards and PostgreSQL prediction logs.

Every concept here transfers directly to real EKS or GKE clusters. The only difference in production is that Terraform targets cloud resources instead of local Helm releases.

Ready to quiz yourself?