Multi-Cluster GitOps with k3d and ArgoCD

Run a hub-and-spoke GitOps pipeline locally using k3d, ArgoCD, and Kustomize.

Introduction

30 Second Summary

When engineering teams deploy code, they often juggle multiple environments like staging and production. Keeping configurations in sync across those environments without someone manually running kubectl commands is the exact problem that GitOps solves.

In this project, you will build a local multi-cluster GitOps pipeline using k3d and ArgoCD on your Windows machine. You'll spin up three Kubernetes clusters, connect them through a hub-and-spoke architecture, and watch ArgoCD automatically deploy different configurations to each environment from a single Git commit.

What You'll Build

You'll push a commit to a Git repo and watch your ArgoCD dashboard light up as it auto-deploys different configurations to staging and production clusters, all running locally on your machine with zero manual intervention.

By the end of this project, you'll have:

  • A hub-and-spoke cluster architecture where a central ArgoCD dashboard shows the real-time health of all applications across staging and production.
  • A fully automated staging environment that deploys every Git push with no manual commands.
  • A production environment with Kustomize overlays delivering higher replica counts and resource limits, proving you can configure per-environment differences declaratively.
  • Proof that Git is the single source of truth: no kubectl apply, no manual intervention, just commits that trigger automatic reconciliation.
  • Secret Mission: Implement the App-of-Apps pattern and self-healing so a single root Application bootstraps your entire multi-cluster delivery, and any manual drift is automatically reverted.

Are there any prerequisites?

You should be comfortable with Docker and basic Kubernetes concepts like pods, deployments, and services.

This project assumes you already have Docker Desktop, kubectl, Chocolatey, and VS Code installed on your Windows machine. Step 1 handles the rest.

Before We Start

Before we dive in, let's lock in what you're about to build and why it matters.

Install Tools and Create Clusters

Your GitOps pipeline needs two tools that are not on your machine yet. k3d spins up lightweight Kubernetes clusters inside Docker containers, and the ArgoCD CLI lets you interact with the GitOps controller that will manage deployments across those clusters.

Without both installed, you cannot create the multi-cluster environment that forms the backbone of this project. In this step, you will install both tools, then spin up three interconnected clusters on a shared Docker network.

In this step, get ready to:
  • Install k3d for creating lightweight Kubernetes clusters.
  • Install the ArgoCD CLI for managing the GitOps controller.
  • Create three interconnected clusters (hub, staging, production) on a shared Docker network.
Install k3d via Chocolatey

k3d is available through Chocolatey, the package manager you already have installed. You need an elevated (Administrator) PowerShell to install packages with Chocolatey.

  • Press the Windows key to open your search bar.
  • Type PowerShell, then right-click Windows PowerShell and select Run as administrator.
  • Install k3d by running this command:
choco install k3d

What does this command do?

Chocolatey downloads the k3d binary and places it on your system PATH so you can run it from any terminal session.

Chocolatey will ask for confirmation before running the install script. Type Y and press Enter when prompted.

  • Verify the installation by running:
k3d version

Check your output against the tabs below.

✔️ I see version 5.9.0 or higher

k3d is installed and ready. Move on to the next verification below.

ⓧ I see an older version

You have an outdated version of k3d. Upgrade it by running:

choco upgrade k3d

Run k3d version again to confirm you see v5.9.0 or later.

ⓧ Command not found

PowerShell cannot find the k3d binary. Close this PowerShell window and open a new elevated PowerShell session. The PATH update from Chocolatey only takes effect in new terminals.

If it still fails, re-run choco install k3d and look for error messages in the output.

  • Confirm kubectl is still working by running:
kubectl version --client

You should see a client version number printed. This confirms your existing kubectl installation is functional.

kubectl command not found?

If kubectl is not recognized, it may have been removed or its PATH entry was lost. Reinstall it via Chocolatey with choco install kubernetes-cli and reopen your terminal.

help me fix kubectl not being found in my PATH on Windows

Install the ArgoCD CLI via PowerShell

The ArgoCD CLI lets you log into the ArgoCD server, register clusters, and trigger syncs from your terminal. You will download the Windows binary directly from the ArgoCD GitHub releases.

  • Download the latest ArgoCD CLI binary by running these commands:
$version = (Invoke-RestMethod https://api.github.com/repos/argoproj/argo-cd/releases/latest).tag_name
$url = "https://github.com/argoproj/argo-cd/releases/download/" + $version + "/argocd-windows-amd64.exe"
Invoke-WebRequest -Uri $url -OutFile argocd.exe

What do these commands do?

  • The first line fetches the latest release tag from GitHub's API and stores it in $version.
  • The second line builds the download URL for the Windows binary.
  • The third line downloads the binary to your current directory as argocd.exe.

The download may take a moment depending on your internet connection. You will see a progress bar in the terminal.

  • Create a tools directory and move the binary there by running:
New-Item -ItemType Directory -Force -Path C:\tools
Move-Item .\argocd.exe C:\tools\argocd.exe
  • Add C:\tools to your user PATH so you can run argocd from anywhere:
[Environment]::SetEnvironmentVariable("Path", "$env:Path;C:\tools", "User")
$env:Path += ";C:\tools"

Why two lines?

The first line persists the PATH change permanently for future sessions. The second line updates your current session so you can verify immediately without reopening the terminal.

  • Verify the ArgoCD CLI is accessible by running:
argocd version --client

Check your output against the tabs below.

✔️ I see a version number

The ArgoCD CLI is installed and on your PATH. You are ready to move on to cluster creation.

ⓧ Command not found

PowerShell cannot find argocd.exe. Confirm the file exists at C:\tools\argocd.exe by running:

Test-Path C:\tools\argocd.exe

If it returns True, your PATH was not updated in this session. Close this PowerShell window and open a new elevated PowerShell. Then re-run argocd version --client.

If it returns False, the Move-Item command did not succeed. Re-run the download and move commands from above.

Create the multi-cluster environment

Your GitOps architecture uses three clusters that communicate over a shared Docker network. The hub cluster will run ArgoCD as the management control plane. The staging and production clusters are spoke clusters that receive deployments.

  • Create the shared Docker network by running:
docker network create gitops-net

Docker prints a long hash string. That confirms the network was created.

  • Create all three k3d clusters on the shared network by running these commands:
k3d cluster create hub --network gitops-net --api-port 6443
k3d cluster create staging --network gitops-net --api-port 6444
k3d cluster create prod --network gitops-net --api-port 6445

What is happening here?

Each command creates a lightweight Kubernetes cluster running inside Docker containers. The --network gitops-net flag attaches every cluster to the same Docker bridge network so they can reach each other by internal hostname.

The --api-port flag assigns a unique port for each cluster's Kubernetes API so they do not conflict on your machine. Each cluster takes 30 to 60 seconds to create. You will see INFO log messages as k3d pulls images and starts containers.

Cluster creation failed?

  • If you see a port conflict error, another process is using that port. Stop it or choose a different --api-port value (e.g. 6543, 6544, 6545).
  • If Docker reports a connection error, confirm Docker Desktop is running. Open Docker Desktop and wait for the engine to finish starting.
  • If you see a network not found error, re-run docker network create gitops-net before retrying the cluster commands.

help me troubleshoot k3d cluster creation failures on Windows with Docker Desktop

Before you check your contexts, what do you expect the output to look like? How many context names should appear?

  • Verify all three cluster contexts exist by running:
kubectl config get-contexts -o name

You should see three names in the output: k3d-hub, k3d-staging, and k3d-prod. If you had other clusters previously, those contexts will also appear. The important thing is that all three new ones are listed.

Your local infrastructure is live. Three Kubernetes clusters are running inside Docker, connected by a shared network that will let ArgoCD on the hub reach the staging and production clusters. Next up, you will install ArgoCD on the hub cluster and register the spokes.

Bootstrap ArgoCD on the Hub Cluster

Your three clusters are running, but right now they are isolated. There is no controller watching a Git repo and pushing state to them.

The hub cluster is your management control plane. Without ArgoCD running here, there is no GitOps controller to reconcile state across clusters. In this step, you install ArgoCD on the hub and connect both spoke clusters so the dashboard shows your entire fleet.

In this step, get ready to:
  • Install ArgoCD on the hub cluster and confirm all pods are healthy.
  • Access the ArgoCD dashboard and register the staging and production spoke clusters.
Install ArgoCD into the hub cluster

ArgoCD needs its own namespace and a set of CRDs, controllers, and a server deployment. You will point kubectl at the hub cluster and apply the official install manifest.

  • Switch your kubectl context to the hub cluster by running this command:
kubectl config use-context k3d-hub
  • Create the argocd namespace by running:
kubectl create namespace argocd

You should see a confirmation message that the namespace was created.

  • Install ArgoCD v3.4.5 by applying the official manifest:
kubectl apply -n argocd --server-side --force-conflicts -f https://raw.githubusercontent.com/argoproj/argo-cd/v3.4.5/manifests/install.yaml

What does this command do?

This downloads and applies all ArgoCD resources (CRDs, deployments, services, RBAC) into the argocd namespace on your hub cluster.

  • The --server-side flag is required because some ArgoCD CRDs exceed the 262KB annotation size limit that client-side apply imposes.
  • The --force-conflicts flag allows taking ownership of previously managed fields so the install succeeds cleanly.

You will see a long list of resources being created. This is expected.

  • Wait for all ArgoCD pods to reach the Ready state by running:
kubectl wait --for=condition=Ready pods --all -n argocd --timeout=120s

This command blocks until every pod in the argocd namespace reports Ready. It prints a success line for each pod that meets the condition.

Pods not becoming Ready?

  • Make sure Docker Desktop is running and has enough memory allocated (at least 4 GB recommended for three clusters).
  • Run kubectl get pods -n argocd to see which pod is stuck. Common causes are image pull errors when your network is slow.
  • If the timeout expires, run the wait command again. Sometimes pods just need more time on a first pull.

Still stuck? Help me debug ArgoCD pods not becoming Ready on my k3d hub cluster.

Access the ArgoCD dashboard and register spoke clusters

ArgoCD is installed, but its web UI and API are only reachable inside the cluster. You need a port-forward to expose it on your machine, then you will authenticate and register the staging and production clusters.

  • Open a second PowerShell terminal. You need a separate terminal because the port-forward command runs continuously and occupies the window.
  • In that second terminal, start the port-forward by running:
kubectl port-forward svc/argocd-server -n argocd 8080:443

Why a separate terminal?

This command keeps running and forwards traffic from localhost:8080 to the ArgoCD server inside the cluster. It will occupy the terminal until you press Ctrl+C. All remaining commands in this step run in your original terminal.

  • Switch back to your original terminal. Retrieve the auto-generated admin password by running:
argocd admin initial-password -n argocd

Copy the password from the output. You will need it in the next command.

  • Log in to ArgoCD via the CLI by running this command, replacing your-password-here with the password you just copied:
argocd login localhost:8080 --username admin --password your-password-here --insecure

What does --insecure do?

ArgoCD generates a self-signed TLS certificate by default. The --insecure flag tells the CLI to skip certificate verification. This is safe for a local setup.

You should see a confirmation that login succeeded and the context was updated.

  • Register the staging cluster with ArgoCD by running:
argocd cluster add k3d-staging --yes
  • Register the production cluster by running:
argocd cluster add k3d-prod --yes

What does cluster registration do?

The argocd cluster add command installs a ServiceAccount called argocd-manager into the kube-system namespace of the target cluster. ArgoCD on the hub uses that service account to deploy workloads remotely.

Because your clusters share the gitops-net Docker network, the hub reaches them via internal hostnames like https://k3d-staging-server-0:6443. No public networking needed.

Before you verify, what do you think argocd cluster list will show? How many clusters should appear?

  • Verify all clusters are registered by running:
argocd cluster list

You should see 3 clusters listed: the hub itself (shown as in-cluster), k3d-staging, and k3d-prod.

  • Open your browser and navigate to https://localhost:8080.
  • Accept the self-signed certificate warning when your browser shows it.
  • Log in with username admin and the password you retrieved earlier.

You should see the ArgoCD dashboard. It will be empty (no applications yet), but the fact that it loads and you can log in proves the control plane is alive.

Cannot reach localhost:8080?

  • Confirm the port-forward is still running in your second terminal. If it stopped, re-run kubectl port-forward svc/argocd-server -n argocd 8080:443.
  • Check that no other process is using port 8080. Run netstat -an | findstr 8080 to see what is listening.
  • Make sure your kubectl context is still k3d-hub. Run kubectl config current-context to confirm.

Still stuck? Help me troubleshoot the ArgoCD dashboard not loading at localhost:8080.

Your GitOps control plane is live. Next up, you will create a Git repository and deploy your first application to staging through ArgoCD.

Deploy to Staging via GitOps

Your ArgoCD hub cluster is alive, the dashboard is accessible, and both spoke clusters are registered. Now it is time to see the core GitOps loop in action.

In a GitOps workflow, a Git repository is the single source of truth for your infrastructure. You declare what you want in Git, and ArgoCD reconciles the cluster to match. No kubectl apply by hand. Just commits.

In this step, get ready to:
  • Create a GitOps repository with base Kubernetes manifests and a staging overlay.
  • Define an ArgoCD Application that points at your staging environment.
  • Sync the application and verify nginx deploys to the staging cluster.
Create the GitOps repository structure

Every GitOps pipeline starts with a repository that holds your desired state. You will create a Kustomize-based folder structure that separates base resources from environment-specific overlays.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create your project folder, move into it, and initialize a Git repo by running these commands:
mkdir gitops-demo
cd gitops-demo
git init
  • Create the directory structure for base manifests, environment overlays, and ArgoCD apps by running:
mkdir base, overlays/staging, overlays/production, argocd-apps
  • Confirm the directories exist by running:
ls

You should see four directories listed: argocd-apps, base, and overlays.

Don't see all the folders?

Make sure you ran the mkdir command from inside the gitops-demo folder. Run pwd to check your current location.

help me fix my directory structure for the GitOps project

Now you will create the base Kubernetes manifests that define an nginx deployment. These base files are shared across all environments.

  • Open the gitops-demo folder in VS Code. Press Ctrl+Shift+P to open the command palette, type File: Open Folder, and select the gitops-demo folder on your Desktop.
  • Create a new file at base/deployment.yaml and paste the following:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.27
        ports:
        - containerPort: 80

What does this manifest do?

  • Declares a Deployment named nginx with 1 replica.
  • Uses the nginx:1.27 container image and exposes port 80 inside the pod.
  • The selector and labels connect the Deployment to its pods using the app: nginx label.
  • Create a new file at base/service.yaml and paste the following:
apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  selector:
    app: nginx
  ports:
  - port: 80
    targetPort: 80
  type: ClusterIP

What does this Service do?

A ClusterIP Service gives the nginx pods a stable internal address. Other pods in the same cluster can reach nginx on port 80 through this Service name.

  • Create a new file at base/kustomization.yaml and paste the following:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml

What is a kustomization file?

This file tells Kustomize which resources belong to this layer. It acts as a manifest listing. ArgoCD detects kustomization.yaml automatically and uses Kustomize to render the final manifests.

  • Save all three files.
  • Verify the base layer renders correctly by running this in your terminal:
kubectl kustomize base/

You should see the combined YAML output showing both the Deployment and Service resources.

Seeing an error from kustomize?

Check that base/kustomization.yaml lists deployment.yaml and service.yaml under resources. File names must match exactly, including the .yaml extension.

help me fix my Kustomize base layer

Now create the staging overlay that customizes the base for your staging environment.

  • Create a new file at overlays/staging/kustomization.yaml and paste the following:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
namespace: demo-app

What does the namespace field do?

The namespace: demo-app line tells Kustomize to add namespace: demo-app to every resource it renders. This isolates your staging workload in its own namespace without modifying the base files.

The resources: [../../base] reference pulls in the Deployment and Service from the base layer.

  • Save the file and verify the staging overlay renders by running:
kubectl kustomize overlays/staging/

You should see the same Deployment and Service, but now every resource includes namespace: demo-app in its metadata.

Create the ArgoCD Application for staging

An ArgoCD Application resource tells the controller what to deploy, where to find it in Git, and which cluster to target. You will create one for staging with manual sync enabled.

  • Create a new file at argocd-apps/staging-app.yaml and paste the following:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: staging-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/[[GITHUB_USERNAME="your-github-username"]]/gitops-demo.git
    targetRevision: HEAD
    path: overlays/staging
  destination:
    server: https://k3d-staging-server-0:6443
    namespace: demo-app
  syncPolicy: {}

What do these fields mean?

  • repoURL points to your GitHub repo where the manifests live.
  • path: overlays/staging tells ArgoCD which directory contains the Kustomize overlay to render.
  • destination.server uses the Docker-internal hostname of the staging cluster. ArgoCD reaches the spoke cluster through the shared gitops-net network.
  • syncPolicy: {} means manual sync only. ArgoCD will detect drift but will not auto-correct it until you explicitly trigger a sync.

Why manual sync first?

Starting with manual sync lets you observe ArgoCD's detection and reconciliation as separate steps. You will enable automated sync in the next step after seeing how the loop works end to end.

  • Save the file.
  • Before ArgoCD can read your manifests, push them to a GitHub repository. Create a new repository called gitops-demo on GitHub (public is fine).
  • Commit and push your files by running:
git add .
git commit -m "Initial GitOps structure"
git remote add origin https://github.com/[[GITHUB_USERNAME="your-github-username"]]/gitops-demo.git
git branch -M main
git push -u origin main

Why push to GitHub?

ArgoCD runs inside a Docker container and cannot access your local filesystem. By hosting your manifests on GitHub, ArgoCD can pull from the repository URL and detect changes on every poll.

Push rejected or authentication error?

Make sure the GitHub repo exists and your Git credentials are configured. If using HTTPS, GitHub requires a Personal Access Token instead of your password.

help me push my gitops-demo repo to GitHub

✔️ Awesome, I've got everything!

Great. Make sure all files are saved and the push to GitHub completed without errors.

ⓧ I'd like to double check the full code

Here is each file you should have:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.27
        ports:
        - containerPort: 80
apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  selector:
    app: nginx
  ports:
  - port: 80
    targetPort: 80
  type: ClusterIP
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
namespace: demo-app
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: staging-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/[[GITHUB_USERNAME="your-github-username"]]/gitops-demo.git
    targetRevision: HEAD
    path: overlays/staging
  destination:
    server: https://k3d-staging-server-0:6443
    namespace: demo-app
  syncPolicy: {}
Apply the staging Application and trigger the first sync

With your manifests on GitHub, you can now tell ArgoCD to start watching the staging overlay. ArgoCD will detect the desired state and wait for you to trigger the sync.

  • Confirm you are on the hub cluster context by running:
kubectl config use-context k3d-hub
  • Apply the staging Application to the ArgoCD namespace by running:
kubectl apply -f argocd-apps/staging-app.yaml -n argocd

You should see application.argoproj.io/staging-app created in the output.

  • Open the ArgoCD dashboard at https://localhost:8080 in your browser.

You should see staging-app appear with an OutOfSync status. This means ArgoCD has compared the desired state in Git to the current state on the staging cluster and found a difference.

What does OutOfSync mean?

ArgoCD continuously compares what Git declares (your manifests) to what actually runs on the cluster. When they differ, the app shows as OutOfSync. Since the staging cluster has no workloads yet, everything in your repo is new.

Before syncing, you need to enable namespace creation for this app. The staging cluster does not have a demo-app namespace yet, and ArgoCD will not create it unless told to.

  • Tell ArgoCD to create the namespace during sync by running:
argocd app set staging-app --sync-option CreateNamespace=true

Why set this on the CLI instead of in the YAML?

You are deliberately keeping the YAML file minimal (syncPolicy: {}) so you can see what happens when this option is missing in the next step. Setting it via the CLI applies it to the live Application without changing your Git file.

Before you trigger the sync, what do you think will happen? Will ArgoCD create the namespace and deploy nginx, or will something fail?

  • Sync the staging app by running:
argocd app sync staging-app

ArgoCD creates the demo-app namespace on the staging cluster, deploys the nginx Deployment and Service, and reports the sync as successful.

  • Verify the pods are running on the staging cluster by running:
kubectl --context k3d-staging get pods -n demo-app

You should see one nginx pod in Running state.

  • Return to the ArgoCD dashboard at https://localhost:8080. Confirm that staging-app now shows as Synced and Healthy (green).

App still showing OutOfSync or SyncFailed?

If you see an error about the namespace not existing, make sure you ran the argocd app set staging-app --sync-option CreateNamespace=true command before syncing.

If ArgoCD cannot reach your repository, verify the repoURL in staging-app.yaml matches your actual GitHub repo URL, and that the repo is public.

help me fix my ArgoCD staging-app sync failure

You just deployed an application to a remote cluster without ever running kubectl apply against it. Next up, you will promote this same pattern to production and discover what happens when a critical sync option is missing.

Promote to Production and Fix the Sync Failure

Your staging environment is deploying smoothly through GitOps. Now you'll promote the same application to production with stronger settings: more replicas and resource limits via a Kustomize overlay.

You'll quickly discover that what worked for staging does not work out of the box for a fresh cluster. This is a real-world GitOps failure that enterprise teams encounter when scaling across environments. You'll diagnose it and apply the fix.

In this step, get ready to:
  • Create a production Kustomize overlay with higher replicas and resource limits.
  • Trigger a sync that fails due to a missing namespace.
  • Fix the Application and enable automated sync with self-healing.
Create the production overlay and Application

Production needs a different configuration than staging. You'll use Kustomize's JSON patch format to increase replicas and add resource limits without touching the shared base manifests.

  • Create the production overlay by opening overlays/production/kustomization.yaml in VS Code and pasting the following:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
namespace: demo-app
patches:
  - target:
      kind: Deployment
      name: nginx
    patch: |
      - op: replace
        path: /spec/replicas
        value: 2
      - op: add
        path: /spec/template/spec/containers/0/resources
        value:
          limits:
            cpu: 100m
            memory: 128Mi
          requests:
            cpu: 50m
            memory: 64Mi

What does this overlay do?

This overlay references the same base as staging but applies two JSON patches to the nginx Deployment.

  • The replace operation bumps replicas from 1 to 2, giving production redundancy.
  • The add operation injects CPU and memory limits so the pod cannot consume unbounded resources on the cluster.
  • Save overlays/production/kustomization.yaml.
  • Confirm the file exists by running:
ls overlays/production/

You should see kustomization.yaml listed in the output.

Now create the ArgoCD Application that deploys this overlay to the production cluster. This follows the same pattern you used for staging.

  • Create argocd-apps/prod-app.yaml with the following content:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: prod-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/your-username/gitops-demo.git
    targetRevision: HEAD
    path: overlays/production
  destination:
    server: https://k3d-prod-server-0:6443
    namespace: demo-app
  syncPolicy: {}

What does this Application define?

This tells ArgoCD to watch the overlays/production path in your Git repo and deploy whatever Kustomize renders to the k3d-prod cluster in the demo-app namespace.

The syncPolicy: {} means manual sync for now. You'll upgrade this after the first deployment succeeds.

  • Save argocd-apps/prod-app.yaml.
  • Commit both new files by running:
git add .
git commit -m "Add production environment"
  • Apply the production Application to your hub cluster by running:
kubectl apply -f argocd-apps/prod-app.yaml -n argocd

You should see application.argoproj.io/prod-app created in the output.

Don't see the Application created?

  • Confirm your context is k3d-hub by running kubectl config current-context.
  • Check that argocd-apps/prod-app.yaml is saved and the YAML indentation is correct.

Still stuck? Help me troubleshoot why my kubectl apply for prod-app.yaml is failing on the hub cluster.

Trigger the sync and observe the failure

The Application is now registered in ArgoCD but has not synced yet. If you check the ArgoCD dashboard at https://localhost:8080, you'll see prod-app appearing as OutOfSync.

Before you trigger the sync, think about what will happen. The production cluster has no demo-app namespace yet. Do you think the sync will succeed or fail?

  • Trigger the sync by running:
argocd app sync prod-app

The sync fails. You'll see output indicating that the namespace "demo-app" does not exist on the production cluster.

  • Open the ArgoCD dashboard at https://localhost:8080 and click on prod-app.
  • Confirm you see the app status as SyncFailed with an error mentioning the missing namespace.

Why did this fail when staging worked?

This is intentional. When you synced staging in the previous step, ArgoCD created the namespace as part of the manual sync process. But without the CreateNamespace=true sync option explicitly set, ArgoCD is not guaranteed to create a missing namespace on every cluster.

This is the exact failure mode enterprise teams hit when copying Application definitions across environments. The fix is a single sync option that tells ArgoCD to create the destination namespace if it does not already exist.

Fix the Application with CreateNamespace=true and enable automated sync

Now that you've seen the failure, you'll fix both Applications at once. You'll add CreateNamespace=true to ensure namespaces are created automatically. You'll also enable automated sync with self-healing so ArgoCD continuously reconciles drift.

  • Replace the entire contents of argocd-apps/prod-app.yaml with:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: prod-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/your-username/gitops-demo.git
    targetRevision: HEAD
    path: overlays/production
  destination:
    server: https://k3d-prod-server-0:6443
    namespace: demo-app
  syncPolicy:
    automated:
      enabled: true
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

What changed?

  • CreateNamespace=true tells ArgoCD to create the demo-app namespace if it does not exist on the target cluster.
  • automated.enabled: true means ArgoCD syncs automatically whenever it detects drift between Git and the live cluster.
  • prune: true removes resources from the cluster if they are deleted from Git.
  • selfHeal: true reverts any manual changes someone makes directly on the cluster.
  • Save argocd-apps/prod-app.yaml.
  • Now update argocd-apps/staging-app.yaml with the same automated sync policy. Replace its entire contents with:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: staging-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/your-username/gitops-demo.git
    targetRevision: HEAD
    path: overlays/staging
  destination:
    server: https://k3d-staging-server-0:6443
    namespace: demo-app
  syncPolicy:
    automated:
      enabled: true
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
  • Save argocd-apps/staging-app.yaml.
  • Commit the fix by running:
git add .
git commit -m "Fix: add CreateNamespace and enable auto-sync"
  • Re-apply both Application manifests to the hub cluster by running:
kubectl apply -f argocd-apps/ -n argocd

You should see both staging-app and prod-app configured.

  • Now trigger the production sync again by running:
argocd app sync prod-app

This time, ArgoCD creates the demo-app namespace on the production cluster and deploys 2 nginx replicas with resource limits.

Sync still failing?

  • Verify the YAML indentation in prod-app.yaml. The syncOptions key must be at the same level as automated, both nested under syncPolicy.
  • Confirm you re-applied with kubectl apply -f argocd-apps/ -n argocd after saving. The cluster needs the updated manifest.
  • Check that the Git repo URL in your Application matches your actual repository.

Need more help? Help me fix my prod-app sync failure after adding CreateNamespace=true.

Time to verify the full end state. What do you expect to see on the production cluster now?

  • Check the production pods by running:
kubectl --context k3d-prod get pods -n demo-app

You should see two nginx pods in Running state. Production is live with 2 replicas.

  • Open the ArgoCD dashboard at https://localhost:8080.
  • Confirm both staging-app and prod-app show as Synced and Healthy (green).

✔️ Awesome, I've got everything!

Great. Double check you've saved both files in the argocd-apps/ directory.

ⓧ I'd like to double check the full code

Here are the final versions of the files you created or edited in this step. Compare each one against yours.

argocd-apps/prod-app.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: prod-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/your-username/gitops-demo.git
    targetRevision: HEAD
    path: overlays/production
  destination:
    server: https://k3d-prod-server-0:6443
    namespace: demo-app
  syncPolicy:
    automated:
      enabled: true
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

argocd-apps/staging-app.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: staging-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/your-username/gitops-demo.git
    targetRevision: HEAD
    path: overlays/staging
  destination:
    server: https://k3d-staging-server-0:6443
    namespace: demo-app
  syncPolicy:
    automated:
      enabled: true
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

overlays/production/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
namespace: demo-app
patches:
  - target:
      kind: Deployment
      name: nginx
    patch: |
      - op: replace
        path: /spec/replicas
        value: 2
      - op: add
        path: /spec/template/spec/containers/0/resources
        value:
          limits:
            cpu: 100m
            memory: 128Mi
          requests:
            cpu: 50m
            memory: 64Mi

You now have two environments running from a single Git repo, each with its own configuration. ArgoCD watches for drift and auto-reconciles. Next up, you'll take this further with the App-of-Apps pattern and prove that self-healing truly enforces Git as the single source of truth.

Secret mission

App-of-Apps and Self-Healing

Managing individual Application manifests does not scale. In this secret mission, you'll implement ArgoCD's App-of-Apps pattern so a single root Application bootstraps your entire multi-cluster delivery. Then you'll prove self-healing works by manually breaking something and watching ArgoCD revert it automatically.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely. This project runs entirely locally using Docker containers, so there are no ongoing costs.

Resources you used:

  • 3 k3d clusters (hub, staging, prod) running as Docker containers.
  • A shared Docker network (gitops-net).
  • The gitops-demo project folder on your Desktop.
  • k3d CLI (installed via Chocolatey).
  • ArgoCD CLI binary (argocd.exe).

Keep everything running

No action needed. Choose this if you're still actively building or experimenting with your GitOps pipeline.

  • Your three k3d clusters persist across Docker Desktop restarts. They continue consuming RAM while running.
  • The ArgoCD port-forward in your separate terminal stays active as long as that terminal is open.
  • k3d and the ArgoCD CLI remain installed and available for future projects.

Pause - I'll come back to this later

Stop the clusters to free up RAM without losing your configuration. You can restart them later and pick up where you left off.

  • Press Ctrl+C in the terminal running the ArgoCD port-forward to stop it.
  • Stop all three clusters by running this command in PowerShell:
k3d cluster stop hub && k3d cluster stop staging && k3d cluster stop prod
  • When you're ready to resume, restart the clusters by running:
k3d cluster start hub && k3d cluster start staging && k3d cluster start prod
  • After restarting, re-run the port-forward command to access the ArgoCD dashboard again:
kubectl port-forward svc/argocd-server -n argocd 8080:443

Delete - I don't want to use this again

Remove all project resources completely. This frees up all RAM and disk space used by the clusters, network, and project files.

  • Press Ctrl+C in the terminal running the ArgoCD port-forward to stop it.
  • Delete all three k3d clusters and the shared Docker network by running:
k3d cluster delete hub && k3d cluster delete staging && k3d cluster delete prod && docker network rm gitops-net
  • Delete the local project folder by running:
Remove-Item -Recurse -Force ~/Desktop/gitops-demo
  • Uninstall k3d via Chocolatey by running this in an elevated PowerShell terminal:
choco uninstall k3d
  • Remove the ArgoCD CLI by deleting the argocd.exe file from the directory you placed it in (e.g., C:\tools).
  • If you added that directory to your PATH solely for ArgoCD, remove it from your system PATH via Settings > System > About > Advanced system settings > Environment Variables.

That's a Wrap!

That's a Wrap!

Nice work! You just built and operated a production-grade multi-cluster GitOps pipeline entirely on your local machine.

You've learned how to:

  • Built a three-cluster Kubernetes environment using k3d and Docker, with a hub cluster managing staging and production spokes through ArgoCD.
  • Deployed applications across environments using Kustomize overlays for environment-specific configuration, delivering different replica counts and resource limits from a single base.
  • Diagnosed and fixed a real-world namespace sync failure, then enabled automated sync with self-healing so Git became the enforced single source of truth.
  • Secret Mission: Implemented the App-of-Apps pattern for scalable multi-cluster bootstrapping and proved drift detection works by watching ArgoCD revert manual changes.

Ready to quiz yourself?