GitOps Canary Deployments with Argo

Build a canary pipeline with automated rollback using Argo CD and Argo Rollouts.

Introduction

30 Second Summary

Every engineering team dreads the same scenario: you push a new version to production, something breaks, and suddenly the whole team is scrambling to roll it back manually. What if your deployment system could detect the problem and fix it on its own?

In this project, you will build a progressive delivery pipeline that automatically deploys new versions of an application in stages, monitors real-time metrics, and rolls back bad releases without any human intervention.

What You'll Build

A complete canary deployment system with automated metric-driven rollback, managed entirely through Git and running on a local Kind cluster.

By the end of this project, you'll have:

  • A GitOps-managed deployment pipeline using Argo CD that syncs Kubernetes resources directly from a Git repository.
  • A canary rollout strategy using Argo Rollouts that progressively shifts traffic from 10% to 100%.
  • An automated rollback system that detects elevated error rates via Prometheus and aborts bad deployments with zero human intervention.
  • Secret Mission: Add latency-based pre-promotion analysis for defense-in-depth.

Are there any prerequisites?

This project assumes you have experience deploying pods and services on Kubernetes, are comfortable with kubectl and YAML manifests, and know Git basics.

You should already have a Kind cluster, kubectl, Helm, Docker, and Git installed on your Linux machine.

Before We Start

In this project, you will build a progressive delivery pipeline that catches bad releases automatically.

Instead of relying on a human watching dashboards after every deploy, you will wire up Argo Rollouts to progressively shift traffic to new versions while Prometheus watches for errors. If something goes wrong, the system rolls back on its own.

In this step, get ready to:
  • Understand what you are building and why it matters.
  • Create a multi-node Kind cluster as your local Kubernetes environment.
  • Verify that kubectl is connected to the cluster.
Create the Kind cluster

Progressive delivery requires multiple replicas, canary pods, and monitoring infrastructure. A single-node cluster would be too constrained for that workload.

You will create a three-node Kind cluster with one control-plane node and two worker nodes. This gives you room for stable pods, canary pods, and the Prometheus stack all running simultaneously.

  • Create the cluster configuration file by running this command:
cat <<EOF > kind-cluster.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
  - role: worker
  - role: worker
EOF

What does this config do?

The kind-cluster.yaml file tells Kind to spin up three Docker containers that act as Kubernetes nodes.

One container runs the control plane (the Kubernetes API server, scheduler, and controller manager). The other two are worker nodes where your application pods will actually run.

  • Create the cluster by running this command:
kind create cluster --name argocd-rollouts --config kind-cluster.yaml

This takes 1-2 minutes. Kind pulls the node image, starts the containers, and configures networking between them.

What does this command do?

  • kind create cluster is the Kind CLI command to provision a new Kubernetes cluster inside Docker containers.
  • --name argocd-rollouts gives the cluster a descriptive name so you can identify it later.
  • --config kind-cluster.yaml points Kind at the configuration file you just created.
  • Verify the cluster is running and kubectl is connected by running:
kubectl get nodes

You should see three nodes listed with a Ready status: one control-plane and two workers.

Not seeing three Ready nodes?

  • Make sure Docker is running. Kind creates the cluster as Docker containers, so Docker must be active.
  • If nodes show NotReady, wait 30 seconds and run kubectl get nodes again. The worker nodes need a moment to join the cluster after the control plane starts.

Help me troubleshoot my Kind cluster nodes not showing Ready status.

✔️ Awesome, I've got everything!

Great. Double check you have saved the kind-cluster.yaml file and that all three nodes are Ready.

ⓧ I'd like to double check the full code

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
  - role: worker
  - role: worker

Your local Kubernetes cluster is up and running. Next up, you will install Argo CD and Argo Rollouts to turn this cluster into a GitOps-managed progressive delivery platform.

Install Argo CD and Argo Rollouts

Before you can deploy anything with GitOps, you need the tools that make it possible. Argo CD is the GitOps engine that watches your Git repository and syncs its contents into your cluster.

Argo Rollouts is the progressive delivery controller that replaces Kubernetes' default all-or-nothing deploys with canary rollouts. Together, they form the backbone of your deployment pipeline.

In this step, get ready to:
  • Install Argo CD v3.4.3 on your Kind cluster.
  • Install the Argo CD CLI and authenticate against the local cluster.
  • Install Argo Rollouts v1.9.0 controller and kubectl plugin.
Install Argo CD v3.4.3 on the Kind cluster

Argo CD runs as a set of controllers inside your Kubernetes cluster. You install it by applying the official manifest, which creates the namespace and all required resources.

  • Install Argo CD v3.4.3 by running these commands:
kubectl create namespace argocd && kubectl apply -n argocd --server-side --force-conflicts -f https://raw.githubusercontent.com/argoproj/argo-cd/v3.4.3/manifests/install.yaml

What does this command do?

This creates the argocd namespace and applies the official Argo CD manifests into it.

The --server-side --force-conflicts flags are required because some Argo CD CRDs exceed the size limit for client-side apply.

  • Wait for all Argo CD pods to reach Running status by running:
kubectl get pods -n argocd -w
  • You should see pods like argocd-server, argocd-repo-server, and argocd-application-controller all showing Running status.
  • Press Ctrl+C to stop watching once all pods are Running.

Pods stuck in Pending or CrashLoopBackOff?

If pods are stuck in Pending, your Kind cluster might not have enough resources. Make sure both worker nodes are running by checking kubectl get nodes.

If you see CrashLoopBackOff, wait a minute and check again. Argo CD pods can take 30-60 seconds to stabilize on initial startup.

Help me troubleshoot my Argo CD pod status issues

Install the Argo CD CLI and authenticate

The Argo CD CLI lets you manage applications, check sync status, and log in to your Argo CD server from the terminal.

  • Download and install the Argo CD CLI by running:
curl -sSL -o argocd-linux-amd64 https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64 && sudo install -m 555 argocd-linux-amd64 /usr/local/bin/argocd && rm argocd-linux-amd64

What does this command do?

This downloads the argocd binary from the latest release, installs it to /usr/local/bin/ with execute permissions, and cleans up the download.

  • Verify the CLI is installed by running:
argocd version --client
  • You should see the Argo CD client version printed in the output.

Command not found?

Make sure /usr/local/bin is in your PATH. Run echo $PATH to check.

Help me fix my Argo CD CLI installation

Now you need to authenticate the CLI against your cluster's Argo CD server. Since Argo CD is not exposed outside the cluster by default, you will use port-forwarding.

  • Start port-forwarding to the Argo CD server in the background by running:
kubectl port-forward svc/argocd-server -n argocd 8080:443 &
  • Retrieve the auto-generated admin password by running:
argocd admin initial-password -n argocd
  • Copy the password that is printed (the first line of output).
  • Log in to Argo CD using the CLI by running:
argocd login localhost:8080 --insecure --username admin --password your-password-here

What is happening here?

The --insecure flag tells the CLI to accept the self-signed TLS certificate that Argo CD generates by default. Replace your-password-here with the actual password you copied from the previous command.

  • You should see a message confirming successful login, similar to:
'admin:login' logged in successfully

Login failed or connection refused?

Make sure the port-forward is still running. Run jobs to check if the background process is alive.

If it was killed, restart it with kubectl port-forward svc/argocd-server -n argocd 8080:443 & and try logging in again.

Help me fix my Argo CD CLI login

Install Argo Rollouts v1.9.0 controller and kubectl plugin

Argo Rollouts extends Kubernetes with a Rollout custom resource that replaces the standard Deployment. It provides canary and blue-green deployment strategies with built-in metric analysis.

  • Install the Argo Rollouts controller by running:
kubectl create namespace argo-rollouts && kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml

What does this command do?

This creates the argo-rollouts namespace and deploys the Argo Rollouts controller into it. The controller watches for Rollout resources and manages progressive traffic shifting.

  • Verify the Argo Rollouts controller is running:
kubectl get pods -n argo-rollouts
  • You should see the argo-rollouts pod in Running status.

Argo Rollouts pod not starting?

Check if the namespace was created successfully with kubectl get ns argo-rollouts. If the pod is stuck, describe it with kubectl describe pod -n argo-rollouts to check for image pull issues or resource constraints.

Help me troubleshoot my Argo Rollouts installation

  • Install the kubectl Argo Rollouts plugin by running:
curl -LO https://github.com/argoproj/argo-rollouts/releases/latest/download/kubectl-argo-rollouts-linux-amd64 && sudo install -m 555 kubectl-argo-rollouts-linux-amd64 /usr/local/bin/kubectl-argo-rollouts && rm kubectl-argo-rollouts-linux-amd64

What does this command do?

This downloads the kubectl-argo-rollouts plugin binary and installs it to /usr/local/bin/. Once installed, you can use it via kubectl argo rollouts to inspect and manage rollouts from the terminal.

  • Verify the plugin is working by running:
kubectl argo rollouts version
  • You should see the Argo Rollouts version printed in the output, confirming the plugin is correctly installed.

Plugin not recognized?

kubectl discovers plugins by looking for executables named kubectl-<plugin> in your PATH. Make sure /usr/local/bin is in your PATH and the binary is named exactly kubectl-argo-rollouts.

Help me fix my kubectl argo rollouts plugin

Your GitOps and progressive delivery controllers are now live on the cluster. Next up, you will deploy Prometheus to give Argo Rollouts the metrics it needs to make automated rollback decisions.

Deploy Prometheus with kube-prometheus-stack

Argo Rollouts can automatically decide whether a new version of your app is healthy or broken. But it needs data to make that decision.

In this step, you will install Prometheus on your cluster using the kube-prometheus-stack Helm chart. This gives your cluster a full metrics pipeline that Argo Rollouts will query later to gate deployments.

In this step, get ready to:
  • Install kube-prometheus-stack via Helm for cluster-wide metrics collection.
  • Verify Prometheus is running and accessible within the cluster.
Install kube-prometheus-stack via Helm

The kube-prometheus-stack bundles Prometheus, Grafana, Alertmanager, and the Prometheus Operator into a single Helm chart. The Prometheus Operator also installs the ServiceMonitor CRD, which is how you will tell Prometheus to scrape your application metrics later.

  • Add the Prometheus community Helm repository and install the chart by running these commands:
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install kube-prom-stack prometheus-community/kube-prometheus-stack --version 86.2.0 --namespace monitoring --create-namespace

What do these commands do?

  • helm repo add registers the Prometheus community chart repository with your local Helm installation.
  • helm repo update fetches the latest chart index so Helm knows about version 86.2.0.
  • helm install deploys the chart with release name kube-prom-stack into a new monitoring namespace. The --create-namespace flag creates that namespace automatically.

This installation takes 1-2 minutes as Kubernetes pulls container images and starts the pods. Wait for the command to return before continuing.

Seeing a timeout or ImagePullBackOff?

If the install hangs or pods show ImagePullBackOff, your Kind cluster might not have enough resources. Ensure Docker has at least 4 GB of memory allocated.

Run kubectl get pods -n monitoring to check individual pod status. If a pod is stuck in Pending, check node capacity with kubectl describe nodes.

help me troubleshoot kube-prometheus-stack pods not starting on my Kind cluster

Verify Prometheus is running and accessible

The chart deploys several components: Prometheus server, Alertmanager, Grafana, kube-state-metrics, node-exporter, and the Prometheus Operator itself. All of them must reach Running status before you can proceed.

  • Check that all monitoring pods are healthy by running:
kubectl get pods -n monitoring
  • You should see 6 or more pods in the monitoring namespace, all showing Running status. Look for pods prefixed with prometheus-kube-prom-stack, alertmanager-kube-prom-stack, kube-prom-stack-grafana, kube-prom-stack-kube-state-metrics, and kube-prom-stack-prometheus-node-exporter.

Why so many pods?

The kube-prometheus-stack is designed for production monitoring. It includes separate components for collecting metrics (Prometheus), alerting (Alertmanager), dashboarding (Grafana), and exporting cluster state (kube-state-metrics and node-exporter).

For this project, you only need Prometheus itself. The other components come along for the ride but will not interfere with the rollout pipeline.

  • Confirm the Prometheus service endpoint by running:
kubectl get svc -n monitoring | grep prometheus
  • Look for the service named kube-prom-stack-kube-prome-prometheus on port 9090. This is the cluster-internal address your AnalysisTemplate will query in a later step.

What is the Prometheus cluster address?

Inside the cluster, other pods can reach Prometheus at http://kube-prom-stack-kube-prome-prometheus.monitoring:9090. This follows Kubernetes DNS naming: <service-name>.<namespace>:<port>.

You will use this exact address in your AnalysisTemplate when you configure Argo Rollouts to query error rate metrics from Prometheus.

Don't see the prometheus service?

Make sure the Helm install completed without errors. Run helm list -n monitoring to confirm the release shows STATUS: deployed.

If the release is missing, re-run the install command. If it shows failed, uninstall with helm uninstall kube-prom-stack -n monitoring and try again.

help me fix my kube-prometheus-stack Helm installation

Your cluster now has a full metrics collection pipeline running. Next up, you will set up the GitOps repository structure and connect Argo CD to manage your application declaratively.

Set Up the GitOps Repository Structure

Your cluster now has Argo CD, Argo Rollouts, and Prometheus all running. The question is: how do you manage all of these Kubernetes resources reliably across deploys?

The answer is GitOps. You store every manifest in a Git repository, and Argo CD watches that repo for changes. When you push a commit, Argo CD detects the diff and syncs your cluster to match. In this step, you will create that repository and connect it to Argo CD.

In this step, get ready to:
  • Create a Git repository with the app-of-apps directory layout.
  • Define the Argo CD Application resource pointing to the repo.
  • Connect Argo CD to the Git repository and verify sync.
Create the Git repository with the app-of-apps directory layout

Argo CD needs a Git repository to watch. You will use an "app-of-apps" pattern where one directory holds the Application CRD definitions and another holds the actual workload manifests.

  • Go to GitHub and create a new public repository named progressive-delivery-demo.
  • Initialize it with a README (this creates the main branch automatically).
  • Move to your Desktop by running this command:
cd ~/Desktop
  • Clone the repository by running this command:
git clone https://github.com/[[YOUR_USERNAME="your GitHub username"]]/progressive-delivery-demo.git
cd progressive-delivery-demo
  • Create the two directories that form the app-of-apps layout by running these commands:
mkdir apps
mkdir manifests
  • Confirm the directories exist by running:
ls

You should see apps, manifests, and README.md listed in the output.

What is the app-of-apps pattern?

The apps/ directory holds Argo CD Application definitions. These tell Argo CD where to look for actual Kubernetes manifests.

The manifests/ directory holds the workload resources (Rollouts, Services, AnalysisTemplates). Argo CD watches this path and syncs everything in it to the cluster automatically.

Now create the first manifest that seeds the repository. This is a simple Namespace resource that Argo CD will create in the cluster.

  • Create the namespace manifest by running this command:
cat > manifests/namespace.yaml << 'EOF'
apiVersion: v1
kind: Namespace
metadata:
  name: demo
EOF

What does this manifest do?

This defines a demo namespace where all your application workloads will live. Argo CD will create this namespace automatically when it syncs.

  • Verify the file was created by running:
cat manifests/namespace.yaml

You should see the YAML content with kind: Namespace and name: demo printed to the terminal.

File is empty or missing?

Make sure you are inside the progressive-delivery-demo directory when running the cat > command. Run pwd to confirm your current directory.

help me debug why my namespace.yaml file is empty or not created

Define the Argo CD Application resource pointing to the repo

An Argo CD Application CRD is the bridge between your Git repository and the cluster. It tells Argo CD which repo to watch, which path contains manifests, and where to deploy them.

  • Create the Application manifest by running this command:
cat > apps/argo-cd-app.yaml << 'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: progressive-delivery-demo
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/[[YOUR_USERNAME="your GitHub username"]]/progressive-delivery-demo.git
    targetRevision: main
    path: manifests
  destination:
    server: https://kubernetes.default.svc
    namespace: demo
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
EOF

What does this Application resource do?

  • source.repoURL points Argo CD at your GitHub repo.
  • source.path: manifests tells it to only sync files from the manifests/ directory.
  • syncPolicy.automated enables auto-sync. Argo CD will reconcile the cluster whenever it detects a Git change.
  • prune: true deletes resources from the cluster if you remove them from Git.
  • selfHeal: true reverts manual cluster changes back to match Git.
  • CreateNamespace=true allows Argo CD to create the destination namespace if it does not exist yet.
  • Verify the file by running:
cat apps/argo-cd-app.yaml

You should see the full Application YAML printed with your GitHub username in the repoURL field.

See YOUR_USERNAME still in the file?

You need to replace YOUR_USERNAME with your actual GitHub username before running the cat > command. Delete the file and re-run the command with your username substituted in the repoURL line.

help me fix the repoURL in my Argo CD Application manifest

✔️ Awesome, I've got everything!

Great. Make sure you have saved both files before continuing.

ⓧ I'd like to double check the full code

Here is apps/argo-cd-app.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: progressive-delivery-demo
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/YOUR_USERNAME/progressive-delivery-demo.git
    targetRevision: main
    path: manifests
  destination:
    server: https://kubernetes.default.svc
    namespace: demo
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Here is manifests/namespace.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: demo
Connect Argo CD to the Git repository and verify sync

With both files created, you need to push them to GitHub and then apply the Application resource to the cluster. Argo CD will pick up the Application and start syncing the manifests/ directory.

  • Stage and push all files to GitHub by running these commands:
git add .
git commit -m "Add app-of-apps structure and namespace manifest"
git push origin main

Authentication error on push?

GitHub requires a personal access token (not your password) for HTTPS pushes. Generate one at github.com/settings/tokens with the repo scope, then paste it when prompted for a password.

help me authenticate my git push to GitHub

  • Apply the Application CRD to the cluster by running:
kubectl apply -f apps/argo-cd-app.yaml

You should see output confirming the Application was created in the argocd namespace.

  • Verify that Argo CD has picked up the Application and synced successfully by running:
argocd app get progressive-delivery-demo

You should see the application details including a Sync Status of Synced and Health Status of Healthy.

What just happened?

Argo CD detected the Application resource, connected to your GitHub repo, found manifests/namespace.yaml, and created the demo namespace in the cluster. From now on, every commit you push to the manifests/ directory will be automatically applied to the cluster.

  • Confirm the demo namespace was created in the cluster by running:
kubectl get namespace demo

You should see the demo namespace listed with an Active status.

  • Start the Argo CD UI port-forward so you can view the Application in your browser:
kubectl port-forward svc/argocd-server -n argocd 8080:443
  • Open your browser and navigate to https://localhost:8080.
  • Accept the self-signed certificate warning.
  • You should see the progressive-delivery-demo Application tile showing a green Synced status.

Application shows OutOfSync or Missing?

Make sure you pushed your commits to GitHub before applying the Application. Run git log --oneline to confirm your commit exists, then check git remote -v to confirm the remote URL matches your repoURL in the Application YAML.

Also verify that the repository is public. Argo CD cannot access private repos without additional credential configuration.

help me debug why my Argo CD Application is not syncing

Your GitOps foundation is solid. Every manifest you add to the manifests/ directory will now be synced to the cluster automatically. Next up, you will create the Rollout and AnalysisTemplate that power the canary deployment.

Create the Rollout and AnalysisTemplate

Your GitOps pipeline is connected and syncing. Now it is time to define the actual deployment logic.

In this step, you will create the core manifests that make progressive delivery work. You will define how your application rolls out (canary steps), how it gets traffic (Services), how it measures health (AnalysisTemplate), and how Prometheus discovers it (ServiceMonitor).

In this step, get ready to:
  • Write the Rollout and Service manifests for canary traffic shifting.
  • Write the AnalysisTemplate and ServiceMonitor for metric-driven analysis.
  • Push to Git and confirm the stable deployment runs via Argo CD.
Write the Rollout and Service manifests

A Rollout replaces a standard Kubernetes Deployment. Instead of instantly swapping all pods, it progressively shifts traffic to the new version in configurable weight steps.

  • Create a file called manifests/rollout.yaml in your progressive-delivery-demo repository and paste the following:
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: demo-app
  namespace: demo
spec:
  replicas: 5
  selector:
    matchLabels:
      app: demo-app
  template:
    metadata:
      labels:
        app: demo-app
    spec:
      containers:
        - name: demo-app
          image: argoproj/rollouts-demo:blue
          ports:
            - containerPort: 8080
  strategy:
    canary:
      analysis:
        templates:
          - templateName: success-rate
        startingStep: 1
        args:
          - name: service-name
            value: demo-app-canary.demo.svc.cluster.local
      canaryService: demo-app-canary
      stableService: demo-app-stable
      steps:
        - setWeight: 10
        - pause: {duration: 30s}
        - setWeight: 30
        - pause: {duration: 30s}
        - setWeight: 60
        - pause: {duration: 30s}
        - setWeight: 100
        - pause: {duration: 30s}

What does this manifest do?

The strategy.canary.steps section defines the traffic progression: 10% to 30% to 60% to 100%, with 30-second pauses between each step.

The analysis block attaches a background AnalysisTemplate called success-rate that starts measuring after step 1. If the analysis fails at any point, the entire rollout aborts automatically.

The argoproj/rollouts-demo:blue image is the official Argo Rollouts demo application. The blue tag represents a healthy, stable version with zero errors.

The Rollout needs two Kubernetes Services to route traffic: one for the stable pods and one for the canary pods.

  • Create manifests/services.yaml and paste the following:
apiVersion: v1
kind: Service
metadata:
  name: demo-app-stable
  namespace: demo
spec:
  selector:
    app: demo-app
  ports:
    - port: 80
      targetPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: demo-app-canary
  namespace: demo
spec:
  selector:
    app: demo-app
  ports:
    - port: 80
      targetPort: 8080

Why two Services?

Argo Rollouts manages which pods each Service selects during a canary. The demo-app-stable Service always points to the proven version. The demo-app-canary Service points only to the new candidate pods receiving canary traffic.

The Rollout controller patches the selectors on these Services dynamically as it progresses through the canary steps.

  • Save both files.

YAML indentation errors?

YAML is sensitive to whitespace. Make sure you use spaces (not tabs) and that nested keys are indented by exactly 2 spaces. A misaligned key will cause Argo CD to report a sync error.

help me fix YAML indentation in my Rollout manifest

Write the AnalysisTemplate and ServiceMonitor

The AnalysisTemplate is the decision-maker. It queries Prometheus for the HTTP success rate of the canary and tells Argo Rollouts whether to continue or abort.

  • Create manifests/analysis-template.yaml and paste the following:
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
  namespace: demo
spec:
  args:
    - name: service-name
  metrics:
    - name: success-rate
      interval: 20s
      successCondition: result[0] >= 0.95
      failureLimit: 3
      provider:
        prometheus:
          address: http://kube-prom-stack-kube-prome-prometheus.monitoring:9090
          query: |
            sum(rate(
              http_requests_total{service="{{args.service-name}}",status!~"5.*"}[30s]
            )) /
            sum(rate(
              http_requests_total{service="{{args.service-name}}"}[30s]
            ))

What does this AnalysisTemplate do?

Every 20 seconds, it queries Prometheus for the ratio of non-5xx requests to total requests. If that ratio drops below 0.95 (meaning more than 5% of requests are errors), it counts as a failure.

After 3 consecutive failures (failureLimit: 3), the AnalysisRun is marked as Failed. This triggers an automatic rollback of the entire canary deployment.

The {{args.service-name}} placeholder gets filled in from the Rollout's analysis.args field, which passes demo-app-canary.demo.svc.cluster.local at runtime.

For Prometheus to actually collect metrics from your application, it needs a ServiceMonitor that tells it where to scrape.

  • Create manifests/service-monitor.yaml and paste the following:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: demo-app
  namespace: demo
  labels:
    release: kube-prom-stack
spec:
  selector:
    matchLabels:
      app: demo-app
  endpoints:
    - port: http
      interval: 10s

Why the release label?

The release: kube-prom-stack label is critical. The Prometheus Operator installed by kube-prometheus-stack only picks up ServiceMonitors that carry this label. Without it, Prometheus will never discover your application's metrics endpoint.

  • Save both files.

Not sure about the Prometheus address?

The address http://kube-prom-stack-kube-prome-prometheus.monitoring:9090 is the in-cluster DNS name for the Prometheus service. It follows the pattern <release-name>-kube-prome-prometheus.<namespace>:<port>. You can verify yours by running:

kubectl get svc -n monitoring | grep prometheus

help me find the correct Prometheus service address in my cluster

Push to Git and verify the deployment

With all manifests written, you need to commit and push them. Argo CD will detect the new files in the manifests/ directory and automatically sync them to your cluster.

  • Stage, commit, and push all the new manifests by running these commands:
git add manifests/
git commit -m "Add Rollout, Services, AnalysisTemplate, and ServiceMonitor"
git push origin main
  • Wait about 30 seconds for Argo CD to detect the change, then check the sync status by running:
argocd app get progressive-delivery-demo
  • You should see Status: Synced and Health: Healthy in the output. All resources should show as synced.
  • Confirm the Rollout is running with 5 healthy replicas by running:
kubectl argo rollouts get rollout demo-app -n demo
  • You should see the Rollout in a Healthy state with 5 replicas running the blue image. The status should show stable because this is the initial deployment (no canary steps run on the first deploy).

Why does the first deploy skip canary steps?

Argo Rollouts treats the initial deployment as a baseline. There is no previous stable version to compare against, so it immediately promotes the first version to stable. Canary steps only kick in when you update the image to a new version.

Rollout not showing as Healthy?

If you see Degraded or Progressing status, check if the pods are pulling the image successfully:

kubectl get pods -n demo

If pods show ImagePullBackOff, your Kind nodes may not have internet access or the image name has a typo. Verify the image name matches argoproj/rollouts-demo:blue exactly.

help me debug why my Argo Rollouts demo-app pods are not starting

✔️ Awesome, I've got everything!

Great. Double check you have saved all four manifest files and that your Argo CD app shows Synced and Healthy.

ⓧ I'd like to double check the full code

Here are all the manifests you should have at this point.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: demo-app
  namespace: demo
spec:
  replicas: 5
  selector:
    matchLabels:
      app: demo-app
  template:
    metadata:
      labels:
        app: demo-app
    spec:
      containers:
        - name: demo-app
          image: argoproj/rollouts-demo:blue
          ports:
            - containerPort: 8080
  strategy:
    canary:
      analysis:
        templates:
          - templateName: success-rate
        startingStep: 1
        args:
          - name: service-name
            value: demo-app-canary.demo.svc.cluster.local
      canaryService: demo-app-canary
      stableService: demo-app-stable
      steps:
        - setWeight: 10
        - pause: {duration: 30s}
        - setWeight: 30
        - pause: {duration: 30s}
        - setWeight: 60
        - pause: {duration: 30s}
        - setWeight: 100
        - pause: {duration: 30s}
apiVersion: v1
kind: Service
metadata:
  name: demo-app-stable
  namespace: demo
spec:
  selector:
    app: demo-app
  ports:
    - port: 80
      targetPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: demo-app-canary
  namespace: demo
spec:
  selector:
    app: demo-app
  ports:
    - port: 80
      targetPort: 8080
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
  namespace: demo
spec:
  args:
    - name: service-name
  metrics:
    - name: success-rate
      interval: 20s
      successCondition: result[0] >= 0.95
      failureLimit: 3
      provider:
        prometheus:
          address: http://kube-prom-stack-kube-prome-prometheus.monitoring:9090
          query: |
            sum(rate(
              http_requests_total{service="{{args.service-name}}",status!~"5.*"}[30s]
            )) /
            sum(rate(
              http_requests_total{service="{{args.service-name}}"}[30s]
            ))
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: demo-app
  namespace: demo
  labels:
    release: kube-prom-stack
spec:
  selector:
    matchLabels:
      app: demo-app
  endpoints:
    - port: http
      interval: 10s

Your progressive delivery pipeline is now fully wired. You have a Rollout that shifts traffic in stages, an AnalysisTemplate that monitors success rate via Prometheus, and a ServiceMonitor that ensures metrics flow. Next up, you will deploy a deliberately broken version and watch the automated rollback in action.

Trigger a Bad Deploy and Watch the Automated Rollback

Your progressive delivery pipeline is fully wired. Argo CD manages the manifests, Argo Rollouts controls the canary steps, and Prometheus scrapes metrics that feed the AnalysisTemplate.

Now it is time to prove the system actually catches a bad release. You will deploy a broken version of the application that returns HTTP 500 errors, and watch the entire automated rollback happen without touching a single button.

In this step, get ready to:
  • Deploy a bad version of the application via a Git commit.
  • Watch Argo Rollouts detect elevated error rates through Prometheus.
  • Confirm the Rollout automatically aborts and restores the stable version.
Update the Rollout image to deploy the bad version

In a real GitOps workflow, you never change running infrastructure directly. You commit a change to Git and let Argo CD sync it to the cluster. This is exactly what you will do now.

  • In manifests/rollout.yaml, find the image line:
          image: argoproj/rollouts-demo:blue
  • Replace blue with yellow, so the line now looks like this:
          image: argoproj/rollouts-demo:yellow
  • Commit and push the change by running these commands:
git add manifests/rollout.yaml
git commit -m "Deploy bad v2 (yellow) to trigger rollback"
git push origin main

What does this change do?

The yellow image tag produces HTTP 500 errors on approximately 50% of requests. Since your AnalysisTemplate requires a 95% success rate (result[0] >= 0.95), this version will fail the analysis check within seconds.

Because Argo CD has automated sync enabled with selfHeal: true, it will detect the new commit and apply the updated Rollout manifest to the cluster automatically.

  • Save manifests/rollout.yaml before committing.
  • Wait about 30 seconds for Argo CD to pick up the commit and sync.

Push rejected or sync not happening?

Make sure you committed the file before pushing. Run git status to confirm there are no uncommitted changes.

If Argo CD is not syncing, verify the Application is still connected by checking the Argo CD UI at https://localhost:8080. The app should show Synced or OutOfSync (then syncing).

help me debug why Argo CD is not syncing my Git changes

Watch the canary rollout begin and the analysis detect errors

Once Argo CD syncs the new image, Argo Rollouts creates a canary ReplicaSet with the yellow image and begins stepping through the canary strategy.

  • Watch the rollout progress in real time by running this command:
kubectl argo rollouts get rollout demo-app -n demo --watch

What should I see?

The terminal displays a live tree view of the Rollout and its child resources. You will see the canary ReplicaSet spin up, traffic shift to 10%, and a background AnalysisRun start (prefixed with the α icon).

Within 60-90 seconds, the AnalysisRun queries Prometheus and detects that the success rate is well below 0.95. After 3 consecutive failures (the failureLimit you configured), the Rollout status changes to Degraded and the canary scales back to zero.

  • Keep the watch running until you see the Rollout status change to Degraded and the canary ReplicaSet scale to 0 pods.

Speed up the wait

The AnalysisRun checks every 20 seconds and needs 3 failures to abort. This means the rollback triggers in approximately 60 seconds after the first analysis measurement.

Rollout not aborting?

If the AnalysisRun stays in Running for more than 2 minutes without failing, Prometheus may not be scraping the canary pods. Verify the ServiceMonitor is active by running:

kubectl get servicemonitor -n demo

You should see demo-app listed. If not, check that the release: kube-prom-stack label matches your Prometheus Operator's label selector.

help me debug why my AnalysisRun is not detecting errors from Prometheus

Confirm the automated rollback completed

After the Rollout aborts, all traffic returns to the stable (blue) ReplicaSet. The canary (yellow) pods are scaled down to zero.

  • Press Ctrl+C to stop the watch if it is still running.
  • Check the AnalysisRun results by running this command:
kubectl get analysisrun -n demo

What does this output show?

You should see an AnalysisRun with a status of Failed. This is the automated verdict that triggered the rollback. The AnalysisRun queried Prometheus, found the success rate below 0.95 on three consecutive checks, and told the Rollout controller to abort.

  • Verify the Rollout is back on the stable version by running:
kubectl get rollout demo-app -n demo -o jsonpath='{.status.phase}'
  • You should see Degraded (the desired state in Git still points to yellow, but the running pods are back on blue).
  • Confirm the pods are running the stable image by running:
kubectl get pods -n demo -o jsonpath='{.items[*].spec.containers[0].image}'
  • You should see all pods running argoproj/rollouts-demo:blue. The yellow canary pods are gone.

Why does it say Degraded instead of Healthy?

The Rollout status shows Degraded because the desired state in Git (yellow) does not match what is actually running (blue). The system protected itself by rolling back, but it knows the declared intent was not fulfilled.

In a production workflow, a team member would revert the Git commit to restore the Rollout to Healthy status. The key takeaway is that zero human intervention was needed to stop the bad release from reaching users.

Still seeing yellow pods?

If yellow pods persist, the AnalysisRun may not have reached its failure limit yet. Wait another 30 seconds and re-check. The failureLimit: 3 setting means 3 failed measurements must occur before the abort triggers.

help me understand why my canary pods are still running after the AnalysisRun should have failed

✔️ Awesome, I've got everything!

Great. Double check that manifests/rollout.yaml is saved and committed with the yellow image tag.

ⓧ I'd like to double check the full code

Here is the updated manifests/rollout.yaml after deploying the bad version:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: demo-app
  namespace: demo
spec:
  replicas: 5
  selector:
    matchLabels:
      app: demo-app
  template:
    metadata:
      labels:
        app: demo-app
    spec:
      containers:
        - name: demo-app
          image: argoproj/rollouts-demo:yellow
          ports:
            - containerPort: 8080
  strategy:
    canary:
      analysis:
        templates:
          - templateName: success-rate
        startingStep: 1
        args:
          - name: service-name
            value: demo-app-canary.demo.svc.cluster.local
      canaryService: demo-app-canary
      stableService: demo-app-stable
      steps:
        - setWeight: 10
        - pause: {duration: 30s}
        - setWeight: 30
        - pause: {duration: 30s}
        - setWeight: 60
        - pause: {duration: 30s}
        - setWeight: 100
        - pause: {duration: 30s}

You just proved that your progressive delivery pipeline catches bad releases automatically. The system detected elevated error rates via Prometheus, failed the AnalysisRun, and rolled back to the stable version. No dashboards to watch, no pages to answer. Next up, the Secret Mission will add a latency-based gate for defense-in-depth.

Secret mission

Add Latency-Based Pre-Promotion Analysis

Your error-rate gate catches broken code. But what about code that works, just slowly? In production, latency spikes degrade user experience long before they trigger 5xx errors. Add a second AnalysisTemplate that gates promotion on P99 latency, creating defense-in-depth for your progressive delivery pipeline.

Clean Up Your Resources

Clean Up Your Resources

This project runs entirely on your local Kind cluster, so there are no ongoing cloud costs. Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

Resources you used:

  • Kind cluster (argocd-rollouts) with Argo CD, Argo Rollouts, and kube-prometheus-stack.
  • Argo CD CLI binary at /usr/local/bin/argocd.
  • kubectl-argo-rollouts plugin at /usr/local/bin/kubectl-argo-rollouts.
  • Local project directory (progressive-delivery-demo).
  • GitHub repository (progressive-delivery-demo).

Keep everything running

No action needed. Choose this if you are still actively experimenting with canary deployments or want to explore additional Argo Rollouts strategies.

  • Your Kind cluster will continue running as Docker containers until you stop Docker or restart your machine.
  • The Argo CD CLI and kubectl-argo-rollouts plugin will remain installed at /usr/local/bin/.
  • Your Git repository stays on GitHub for portfolio use.

Pause - I'll come back to this later

Pause the Kind cluster containers to free up CPU and memory while preserving your cluster state.

  • Stop the Kind cluster containers by running:
docker stop $(kind get nodes --name argocd-rollouts)
  • When you are ready to resume, start the containers again by running:
docker start $(kind get nodes --name argocd-rollouts)
  • Your local files, CLI tools, and GitHub repository remain untouched.

Delete - I don't want to use this again

Remove all project resources and start fresh. This deletes the cluster and all Kubernetes resources inside it.

  • Delete the Kind cluster by running:
kind delete cluster --name argocd-rollouts
  • Remove the Argo CD CLI and kubectl-argo-rollouts plugin by running:
sudo rm /usr/local/bin/argocd
sudo rm /usr/local/bin/kubectl-argo-rollouts
  • Remove the Helm repository by running:
helm repo remove prometheus-community
  • Delete your local project directory by running:
cd ~/Desktop
rm -rf progressive-delivery-demo
  • To delete the GitHub repository, go to your repository page on GitHub.
  • Click Settings at the top of the repository.
  • Scroll to the bottom and click Delete this repository under the Danger Zone section.
  • Follow the confirmation prompts to permanently remove the repository.

Nice Work!

Nice Work!

Nice work! You just built a production-grade progressive delivery pipeline with automated metric-driven rollback, all running on your local Kind cluster.

You've learned how to:

  • Deploy and manage Kubernetes resources declaratively with Argo CD, using an Application CRD that auto-syncs from Git.
  • Configure Argo Rollouts for canary traffic shifting (10% to 30% to 60% to 100%) with background Prometheus analysis that automatically aborts a bad deploy.
  • Write an AnalysisTemplate that queries Prometheus for HTTP error rate and gates promotion on a 95% success threshold.
  • Added a latency-based pre-promotion gate using P99 histogram metrics to block slow releases before they reach 100% of traffic.

Ready to quiz yourself?