Kubernetes Disaster Recovery with Velero

Set up dual-destination Kubernetes DR with Velero, MinIO, and AWS S3.

Introduction

30 Second Summary

Production systems fail. Disks corrupt, namespaces get accidentally deleted, and clusters go down. The difference between a minor incident and a catastrophe comes down to one thing: whether you have a tested, multi-destination backup strategy that can restore everything.

In this project, you will build a complete disaster recovery system for Kubernetes using Velero. You will deploy a stateful workload, back it up to two separate storage destinations, simulate a catastrophic failure, and perform a validated restore.

What You'll Build

A production-grade disaster recovery environment on a local kind cluster that mirrors enterprise backup patterns with dual storage destinations and a fully validated restore playbook.

By the end of this project, you'll have:

  • A kind cluster running Cilium CNI with a stateful PostgreSQL workload protected by NetworkPolicies and custom CRDs.
  • Velero configured with dual BackupStorageLocations sending backups to both AWS S3 for offsite recovery and in-cluster MinIO for fast local recovery.
  • A proven disaster recovery playbook where you simulate catastrophic namespace deletion, restore from backup, and validate PostgreSQL data integrity, NetworkPolicy enforcement, and CRD restoration.
  • Secret Mission: Automated scheduled backups with retention policies and a self-validating DR drill script.

Are there any prerequisites?

This project assumes familiarity with Kubernetes concepts (deployments, PVCs, namespaces, CRDs) and that you already have Docker, Helm, kubectl, kind, and the AWS CLI installed and configured.

Step 1 covers installing the Velero CLI and Cilium CLI, which are the only new tools you will set up during the project.

Set Up the DR Environment

Before you can back up anything, you need the infrastructure that makes backups possible.

In this step, you will create a local Kubernetes cluster with Cilium as the CNI, install the Velero CLI, and deploy MinIO as an in-cluster S3-compatible storage backend.

In this step, get ready to:
  • Create a kind cluster with Cilium CNI.
  • Install the Velero CLI.
  • Deploy MinIO in-cluster and create an AWS S3 bucket.
Create a kind cluster with Cilium CNI

A production Kubernetes cluster uses a CNI plugin to manage pod networking and enforce network policies. Cilium is a powerful CNI that uses eBPF to provide high-performance networking and security features like CiliumNetworkPolicies.

To use Cilium with kind, you must disable kind's built-in CNI so Cilium can take over.

  • Move to your Desktop by running this command:
cd ~/Desktop
  • Create a project directory and move into it by running these commands:
mkdir velero-dr-project
cd velero-dr-project
  • Create the kind cluster configuration file by running:
cat <<EOF > kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  disableDefaultCNI: true
  podSubnet: "10.244.0.0/16"
nodes:
  - role: control-plane
  - role: worker
  - role: worker
EOF

What does this config do?

The disableDefaultCNI: true line tells kind not to install its default CNI (kindnet).

The podSubnet defines the CIDR range Cilium will use for pod IP addresses. The cluster has one control-plane node and two workers, which gives you a realistic multi-node environment for testing disaster recovery.

  • Create the kind cluster by running:
kind create cluster --config kind-config.yaml

This takes 1-2 minutes. You should see messages about creating the cluster nodes and setting your kubectl context.

Cluster creation failing?

  • Make sure Docker is running. Kind requires the Docker daemon to create cluster nodes as containers.
  • If you see a port conflict, delete any existing kind cluster first with kind delete cluster and try again.

Help me troubleshoot my kind cluster creation

  • Add the Cilium Helm repository by running:
helm repo add cilium https://helm.cilium.io/
  • Install Cilium into the cluster by running:
helm install cilium cilium/cilium --version 1.19.4 \
  --namespace kube-system \
  --set image.pullPolicy=IfNotPresent \
  --set ipam.mode=kubernetes

What does this command do?

  • It installs Cilium v1.19.4 from the Cilium Helm chart into the kube-system namespace.
  • The image.pullPolicy=IfNotPresent setting avoids unnecessary image pulls if the image is already cached in kind.
  • The ipam.mode=kubernetes tells Cilium to use the Kubernetes-native IP address management, which works well with the podSubnet you configured.
  • Wait for Cilium to become ready (this may take 2-3 minutes) by running:
kubectl wait --for=condition=Ready nodes --all --timeout=300s
  • Verify all nodes are Ready by running:
kubectl get nodes

You should see three nodes (one control-plane, two workers) all showing Ready status.

  • Confirm Cilium is healthy by running:
kubectl -n kube-system get pods -l app.kubernetes.io/name=cilium

You should see Cilium agent pods Running on each node.

Nodes stuck in NotReady?

  • Without a CNI, nodes stay NotReady. Confirm Cilium pods are running with the command above.
  • If Cilium pods are in CrashLoopBackOff, check their logs with kubectl -n kube-system logs -l app.kubernetes.io/name=cilium for details.
  • Ensure you set disableDefaultCNI: true in your kind config. If kindnet is running alongside Cilium, they will conflict.

Help me debug why my kind nodes are stuck in NotReady after installing Cilium

Install Velero CLI

The Velero CLI is the command-line tool you use to create backups, restore them, and manage backup storage locations. You need it installed locally so you can interact with the Velero server component you will deploy later.

  • Download and install the Velero v1.18.1 CLI by running these commands:
curl -LO https://github.com/vmware-tanzu/velero/releases/download/v1.18.1/velero-v1.18.1-darwin-arm64.tar.gz
tar -xzf velero-v1.18.1-darwin-arm64.tar.gz
sudo mv velero-v1.18.1-darwin-arm64/velero /usr/local/bin/velero

What do these commands do?

  • The curl -LO downloads the Velero CLI tarball for macOS (Apple Silicon).
  • The tar -xzf extracts the binary from the archive.
  • The sudo mv places the binary in your PATH so you can run velero from any directory.

Your terminal will prompt for your password when you run the sudo mv command. Enter it to continue.

  • Verify the Velero CLI is installed by running:
velero version --client-only

You should see output showing Version: v1.18.1.

Command not found?

  • If you see command not found, confirm the binary was moved correctly. Run ls /usr/local/bin/velero to check it exists.
  • If you are on an Intel Mac, replace darwin-arm64 with darwin-amd64 in the download and extract commands.
  • If you are on Linux, use linux-amd64 or linux-arm64 depending on your architecture.

Help me install the Velero CLI on my machine

Deploy MinIO in-cluster and create the AWS S3 bucket

Your disaster recovery strategy uses two backup destinations. MinIO provides a fast, local backup target inside the cluster. AWS S3 provides an offsite backup that survives even if the entire cluster is destroyed.

Why two backup destinations?

Enterprise DR strategies never rely on a single backup location. Local backups (MinIO) give you fast recovery times. Offsite backups (S3) protect against total infrastructure failure.

This dual-destination pattern is a core principle of production disaster recovery.

  • Deploy MinIO into the cluster using the Velero-provided example manifest by running:
kubectl apply -f https://raw.githubusercontent.com/vmware-tanzu/velero/v1.18.1/examples/minio/00-minio-deployment.yaml

What does this manifest create?

  • A velero namespace.
  • A MinIO Deployment running a single-node S3-compatible object store.
  • A Service exposing MinIO within the cluster at minio.velero.svc:9000.
  • A setup Job that creates the default velero bucket inside MinIO.
  • Wait for the MinIO pod to be ready by running:
kubectl wait --for=condition=Ready pod -l component=minio -n velero --timeout=120s
  • Verify MinIO is running by checking the pods:
kubectl get pods -n velero

You should see the minio pod with status Running and the minio-setup job with status Completed.

MinIO pod in CrashLoopBackOff?

  • This usually means insufficient disk space. MinIO requires at least 900 MiB of available disk. Check Docker's allocated disk space in Docker Desktop settings.
  • Run kubectl logs -n velero -l component=minio to see the exact error message from the MinIO container.

Help me fix my MinIO pod that's crashing

  • Now create the AWS S3 bucket for offsite backups by running:
aws s3 mb s3://[[BUCKET_NAME="velero-dr-project-your-id"]]

Why a unique bucket name?

S3 bucket names are globally unique across all AWS accounts. Appending a personal identifier (like your initials or a short random string) prevents naming conflicts.

  • Verify the bucket was created by running:
aws s3 ls | grep velero-dr-project

You should see your bucket name listed in the output.

Bucket creation failed?

  • If you see an access denied error, confirm your AWS CLI is configured with valid credentials by running aws sts get-caller-identity.
  • If you get a bucket name conflict, choose a more unique name. Try appending random digits.
  • If you are in a region other than us-east-1, add --region followed by your region to the command.

Help me create an S3 bucket for Velero backups

Your environment is ready. You have a multi-node kind cluster with Cilium providing advanced networking, the Velero CLI installed, MinIO running as a local backup target, and an S3 bucket for offsite backups. Next up, you will deploy the stateful application stack that you will protect with backups.

Deploy the Stateful Application Stack

Your cluster is running with Cilium healthy and Velero's CLI ready to go. Now you need something worth backing up.

In this step, you will deploy a realistic multi-component application stack into a dedicated namespace. This includes a stateful PostgreSQL database with persistent storage, a CiliumNetworkPolicy restricting traffic, and a custom CRD representing an architecture config component. Together these simulate the real-world mix of data, network security, and platform extensions that a DR system must protect.

In this step, get ready to:
  • Deploy PostgreSQL with a PersistentVolumeClaim and insert seed data.
  • Apply a CiliumNetworkPolicy and a custom CRD with a CR instance.
  • Verify all resources are active and queryable.
Deploy PostgreSQL with persistent storage

PostgreSQL is the stateful workload at the heart of this project. You will deploy it into a production namespace with a PersistentVolumeClaim so Velero has real volume data to back up.

  • Create the production namespace by running this command:
kubectl create namespace production
  • Create a file called postgres.yaml and paste the following manifest to define the PVC, Deployment, and Service:
# PersistentVolumeClaim for PostgreSQL data
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-pvc
  namespace: production
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
# PostgreSQL Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres
  namespace: production
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:16
          ports:
            - containerPort: 5432
          env:
            - name: POSTGRES_DB
              value: "appdb"
            - name: POSTGRES_USER
              value: "appuser"
            - name: POSTGRES_PASSWORD
              value: "apppassword"
            - name: PGDATA
              value: "/var/lib/postgresql/data/pgdata"
          volumeMounts:
            - name: postgres-storage
              mountPath: /var/lib/postgresql/data
      volumes:
        - name: postgres-storage
          persistentVolumeClaim:
            claimName: postgres-pvc
---
# Service to expose PostgreSQL within the cluster
apiVersion: v1
kind: Service
metadata:
  name: postgres
  namespace: production
spec:
  selector:
    app: postgres
  ports:
    - port: 5432
      targetPort: 5432

What does this manifest do?

This multi-document YAML creates three resources in the production namespace.

  • A 1Gi PersistentVolumeClaim named postgres-pvc that kind's default storage provisioner will bind automatically.
  • A Deployment running the postgres:16 image with credentials set via environment variables. The PGDATA variable points PostgreSQL to a subdirectory inside the mount to avoid "directory not empty" errors on kind.
  • A ClusterIP Service so other pods can reach PostgreSQL at postgres.production.svc:5432.
  • Apply the manifest by running:
kubectl apply -f postgres.yaml
  • Wait for the PostgreSQL pod to become Ready by running:
kubectl wait --for=condition=Ready pod -l app=postgres -n production --timeout=120s

You should see a message indicating the pod condition was met.

Pod stuck in Pending?

If the pod stays in Pending, the PVC may not be binding. Run kubectl get pvc -n production to check its status. On kind, the default storage class provisions volumes automatically. If it shows Pending, confirm your kind cluster has the standard StorageClass by running kubectl get storageclass.

help me troubleshoot why my PostgreSQL pod is stuck in Pending on kind

Now insert seed data that you will verify after a restore later.

  • Insert a test table and row into PostgreSQL by running:
kubectl exec -n production deploy/postgres -- psql -U appuser -d appdb -c "
CREATE TABLE dr_test (id SERIAL PRIMARY KEY, name VARCHAR(50), created_at TIMESTAMP DEFAULT NOW());
INSERT INTO dr_test (name) VALUES ('velero-backup-test');
SELECT * FROM dr_test;
"

What does this command do?

This uses kubectl exec to run psql inside the running PostgreSQL pod. It creates a table called dr_test with an auto-incrementing ID, a name column, and a timestamp. Then it inserts one row with the value velero-backup-test. The SELECT at the end prints the row so you can confirm it was inserted.

You should see a table with one row showing velero-backup-test in the output.

Seeing a connection refused error?

If you get a connection error, the pod may not be fully ready yet. Wait a few seconds and retry. You can confirm the pod is running with kubectl get pods -n production.

help me fix a connection refused error when running psql inside my PostgreSQL pod on kind

Apply Cilium NetworkPolicies and a custom CRD

In production, databases are never left open to all traffic. A CiliumNetworkPolicy locks down ingress so only authorized pods can reach PostgreSQL. You will also define a custom CRD to represent an architecture config component that Velero must preserve.

  • Create a file called network-policy.yaml and paste this CiliumNetworkPolicy:
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: postgres-ingress
  namespace: production
spec:
  # Select the PostgreSQL pod as the destination
  endpointSelector:
    matchLabels:
      app: postgres
  # Only allow ingress from pods with the 'access: postgres' label
  ingress:
    - fromEndpoints:
        - matchLabels:
            access: postgres
      toPorts:
        - ports:
            - port: "5432"
              protocol: TCP

What does this policy do?

The endpointSelector targets pods with the label app: postgres. The ingress rule only permits traffic from pods that carry the label access: postgres on TCP port 5432. Any pod without that label is denied.

This is the kind of platform-level security config that often gets lost in a naive restore. Velero needs to capture it alongside the workload.

  • Apply the network policy by running:
kubectl apply -f network-policy.yaml

You should see ciliumnetworkpolicy.cilium.io/postgres-ingress created in the output.

Error about unknown resource type?

If kubectl does not recognize CiliumNetworkPolicy, Cilium may not be fully installed. Verify with kubectl get crds | grep cilium. You should see several Cilium CRDs listed. If not, revisit Step 1 and confirm the Cilium Helm install completed.

help me troubleshoot why CiliumNetworkPolicy is not recognized on my kind cluster

Next, define the custom CRD and create an instance of it.

  • Create a file called backuppolicy-crd.yaml and paste the CRD definition:
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: backuppolicies.dr.example.com
spec:
  group: dr.example.com
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                schedule:
                  type: string
                retention:
                  type: string
                destinations:
                  type: array
                  items:
                    type: string
  scope: Namespaced
  names:
    plural: backuppolicies
    singular: backuppolicy
    kind: BackupPolicy
    shortNames:
      - bp

What is a CRD?

A CustomResourceDefinition extends the Kubernetes API with your own resource types. Here you are creating a BackupPolicy kind under the group dr.example.com. Once applied, you can create instances of BackupPolicy just like any built-in Kubernetes resource.

CRDs are cluster-scoped objects. Velero needs a separate cluster-scoped backup to capture them, which you will configure in the next step.

  • Apply the CRD by running:
kubectl apply -f backuppolicy-crd.yaml

You should see customresourcedefinition.apiextensions.k8s.io/backuppolicies.dr.example.com created in the output.

  • Now create an instance of the BackupPolicy CR. Create a file called backup-policy-cr.yaml and paste:
apiVersion: dr.example.com/v1
kind: BackupPolicy
metadata:
  name: main-backup-policy
  namespace: production
spec:
  schedule: "0 */6 * * *"
  retention: "30d"
  destinations:
    - minio-local
    - aws-offsite
  • Apply the CR by running:
kubectl apply -f backup-policy-cr.yaml

You should see backuppolicy.dr.example.com/main-backup-policy created in the output.

Seeing a "no matches for kind" error?

If the CR fails with a "no matches for kind" message, the CRD was not registered yet. Wait a few seconds for the API server to process it and try the apply again. You can verify with kubectl get crds | grep backuppolicies.

help me fix a no matches for kind BackupPolicy error in Kubernetes

Verify the application stack

Before moving on to Velero configuration, confirm that every resource is active and queryable.

  • Confirm PostgreSQL is serving data by running:
kubectl exec -n production deploy/postgres -- psql -U appuser -d appdb -c "SELECT * FROM dr_test;"

You should see your velero-backup-test row in the output.

  • Verify the CiliumNetworkPolicy is active by running:
kubectl get ciliumnetworkpolicies -n production

You should see postgres-ingress listed.

  • Verify the custom CRD and CR exist by running:
kubectl get backuppolicies.dr.example.com -n production

You should see main-backup-policy listed.

Why verify everything now?

These exact outputs become your baseline. After the disaster simulation in Step 5, you will compare the restored state against these results to prove recovery completeness. If anything is missing here, it will be missing from your backup too.

Your application stack is deployed with real data, network security, and platform extensions. Next up, you will install Velero server-side and configure it to back all of this up to two destinations.

Configure Velero with Dual Backup Destinations

You have a running application stack with PostgreSQL, Cilium policies, and custom CRDs. Now you need to protect all of it with a backup system that stores copies in two separate locations.

A single backup destination creates a single point of failure. In this step, you will install the Velero server with a dual-destination strategy. Local backups to MinIO provide fast recovery, while offsite backups to AWS S3 protect against cluster-level failures.

In this step, get ready to:
  • Install the Velero server with MinIO as the default backup storage location.
  • Add AWS S3 as a second backup storage location with its own credentials.
  • Verify both locations report Available status.
Install Velero server with MinIO as the default BSL

The velero install command deploys the server-side components into your cluster. It also configures the default BackupStorageLocation that Velero uses when no other location is specified.

  • Create a credentials file for MinIO by running this command:
cat <<EOF > credentials-velero
[default]
aws_access_key_id = minio
aws_secret_access_key = minio123
EOF

What does this credentials file do?

Velero uses the AWS SDK under the hood, even when talking to MinIO. This file provides the access key and secret key that match the MinIO example deployment's default credentials.

The [default] profile header is required by the AWS SDK credential file format.

  • Install the Velero server with MinIO as the default backup location by running this command:
velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-aws:v1.14.0 \
  --bucket velero \
  --secret-file ./credentials-velero \
  --use-volume-snapshots=false \
  --use-node-agent \
  --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://minio.velero.svc:9000

What do these flags do?

  • --provider aws tells Velero to use the AWS S3 protocol. MinIO is S3-compatible, so this works for both MinIO and real S3.
  • --plugins velero/velero-plugin-for-aws:v1.14.0 installs the AWS plugin that handles S3 communication.
  • --use-volume-snapshots=false disables volume snapshot support since kind does not have a CSI snapshot provider.
  • --use-node-agent deploys a DaemonSet that enables file-system backup of persistent volumes using Kopia.
  • s3ForcePathStyle="true" is required for MinIO. Without it, the AWS SDK uses virtual-host style addressing which MinIO does not support.
  • s3Url=http://minio.velero.svc:9000 points Velero to the in-cluster MinIO service instead of the public AWS S3 endpoint.
  • Wait for the Velero deployment and node-agent to become ready by running:
kubectl -n velero wait --for=condition=Available deployment/velero --timeout=120s
kubectl -n velero rollout status daemonset/node-agent --timeout=120s

You should see messages confirming the deployment condition is met and the DaemonSet is successfully rolled out.

Velero pod stuck in CrashLoopBackOff?

  • Check the Velero pod logs with kubectl -n velero logs deploy/velero. If you see connection errors to MinIO, confirm the MinIO pod is still Running with kubectl -n velero get pods.
  • Verify your credentials-velero file has the exact credentials minio and minio123 with no extra whitespace.

Help me debug my Velero installation on a kind cluster with MinIO

Add AWS S3 as a second BackupStorageLocation

Velero only supports configuring one BSL at install time. To add a second destination, you create a separate credentials secret and use velero backup-location create to register the AWS S3 bucket.

  • Create a credentials file for your AWS account by running:
cat <<EOF > credentials-aws
[default]
aws_access_key_id = your-aws-access-key-id
aws_secret_access_key = your-aws-secret-access-key
EOF

Where do I find my AWS credentials?

These are the same credentials you used to configure the AWS CLI earlier. You can find them in ~/.aws/credentials on your machine, or generate new ones in the AWS IAM console under your user's Security credentials tab.

  • Create a Kubernetes Secret in the velero namespace containing your AWS credentials by running:
kubectl create secret generic -n velero aws-credentials \
  --from-file=cloud=./credentials-aws

What does --from-file=cloud= mean?

This creates a secret named aws-credentials with a key called cloud inside it. The key holds the contents of your credentials file.

You will reference this key name when creating the backup location so Velero knows which data to fetch from the secret.

  • Create the AWS S3 backup storage location by running:
velero backup-location create aws-offsite \
  --provider aws \
  --bucket [[BUCKET_NAME="velero-dr-project-your-id"]] \
  --config region=[[AWS_REGION="your AWS region e.g. us-east-1"]] \
  --credential=aws-credentials=cloud

What does --credential=aws-credentials=cloud mean?

The format is --credential=<secret-name>=<key-within-secret>. This tells Velero to fetch credentials from the cloud key inside the aws-credentials secret.

This keeps the AWS credentials separate from the MinIO credentials used by the default BSL. Each location uses its own secret independently.

You should see a confirmation message that the backup storage location was created successfully.

Seeing a secret not found error?

  • Verify the secret exists with kubectl -n velero get secret aws-credentials. If it is missing, re-run the kubectl create secret command above.
  • Make sure you are in the same directory where you created the credentials-aws file.

Help me troubleshoot adding a second BackupStorageLocation in Velero

Verify both locations are Available

Both backup storage locations need to report Available status before you can run backups against them. Velero periodically validates connectivity to each location.

  • Check the status of both backup locations by running:
velero backup-location get

You should see two entries. The default location points to MinIO and aws-offsite points to your S3 bucket. Both should show Available in the status column.

What if a location shows Unavailable?

Velero validates locations every 60 seconds by default. If you just created the location, wait a moment and run velero backup-location get again.

If it stays Unavailable, check the Velero logs with kubectl -n velero logs deploy/velero for connection or credential errors.

AWS location showing Unavailable?

  • Verify the S3 bucket name and region match what you created in Step 1.
  • Confirm the AWS credentials in your credentials-aws file are correct and the IAM user has S3 permissions.
  • Check that the kind cluster's Docker network has outbound internet access to reach AWS S3.

Help me fix my Velero AWS S3 BackupStorageLocation that shows Unavailable

Your dual-destination backup infrastructure is ready. Next up, you will execute full-scope backups to both locations and verify everything is captured correctly.

Execute Multi-Destination Backups

Your dual backup infrastructure is ready. Both storage locations are Available, and your production namespace is running a stateful workload with network policies and custom resources.

Now it is time to prove that the infrastructure actually works by executing backups to both destinations. You will create four backups total: a namespace-scoped backup and a cluster-scoped backup to each location.

In this step, get ready to:
  • Back up the production namespace (with PVC data) to MinIO.
  • Replicate the same backups to AWS S3 for offsite protection.
  • Verify all four backups completed successfully with no errors.
Back up the production namespace to MinIO

Enterprise backup strategies separate namespace-scoped resources (pods, services, PVCs) from cluster-scoped resources (CRDs, ClusterRoles). This ensures you can restore each layer independently.

  • Create a namespace-scoped backup of the production namespace to MinIO (the default BSL) by running this command:
velero backup create production-ns-minio \
  --include-namespaces production \
  --default-volumes-to-fs-backup \
  --wait

What does this command do?

  • production-ns-minio is the backup name. It identifies where this backup was sent.
  • --include-namespaces production scopes the backup to only the production namespace resources (Deployments, Services, PVCs, CiliumNetworkPolicies, CRs).
  • --default-volumes-to-fs-backup tells Velero to use file-system backup (via Kopia) for all PVC data instead of volume snapshots. This is necessary on kind because no cloud snapshot provider is available.
  • --wait blocks the terminal until the backup finishes, so you know when it completes.
  • Since no --storage-location is specified, Velero sends this backup to the default BSL (MinIO).
  • Once that completes, create a cluster-scoped backup to capture CRD definitions by running:
velero backup create cluster-crds-minio \
  --include-cluster-resources=true \
  --include-namespaces="" \
  --wait

Why a separate cluster-scoped backup?

When you use --include-namespaces to filter to a specific namespace, Velero does not include cluster-scoped resources like CRD definitions by default. A separate backup with --include-cluster-resources=true and an empty namespace filter captures all cluster-wide objects including your backuppolicies.dr.example.com CRD.

This two-backup pattern (namespace + cluster) guarantees you can restore the CRD definition before restoring the CR instances that depend on it.

  • You should see output indicating both backups completed. Confirm they exist by running:
velero backup get

You should see production-ns-minio and cluster-crds-minio both listed with status Completed.

Backup stuck in InProgress or showing PartiallyFailed?

  • If the namespace backup hangs, the node-agent DaemonSet may not be running. Check with kubectl get pods -n velero and confirm you see a node-agent pod on each worker node.
  • If you see PartiallyFailed, run velero backup logs production-ns-minio to see which specific resource caused the issue.
  • Make sure the MinIO pod is still Running in the velero namespace.

Still stuck? Help me troubleshoot my Velero backup to MinIO that is not completing.

Back up the production namespace to AWS S3

The same data now needs to land in your offsite location. This is the core of a multi-destination strategy: if your cluster goes down entirely, the S3 copy is still accessible from any new cluster.

  • Create the namespace-scoped backup targeting the aws-offsite BSL by running:
velero backup create production-ns-s3 \
  --include-namespaces production \
  --default-volumes-to-fs-backup \
  --storage-location aws-offsite \
  --wait
  • Create the cluster-scoped backup to S3 by running:
velero backup create cluster-crds-s3 \
  --include-cluster-resources=true \
  --include-namespaces="" \
  --storage-location aws-offsite \
  --wait

What does --storage-location do?

The --storage-location aws-offsite flag tells Velero to send this backup's tarball and metadata to your AWS S3 bucket instead of the default MinIO BSL. The backup process is identical. Only the destination changes.

In production, you would run these to both locations on every backup cycle. This gives you fast local restores from MinIO and disaster-proof offsite copies on S3.

  • Confirm the S3 backups completed by running:
velero backup get

You should now see four backups listed, all with status Completed.

S3 backup failed with permission error?

  • Verify the aws-credentials secret in the velero namespace contains valid credentials with s3:PutObject and s3:GetObject permissions on your bucket.
  • Check that the bucket region in your BSL config matches where you created the bucket.
  • Run velero backup logs cluster-crds-s3 to see the exact error message from the AWS SDK.

Need help? Help me debug my Velero backup to AWS S3 that failed with a permissions error.

Verify backup completeness

A backup showing Completed status is not enough on its own. You need to verify that the backup actually captured everything: all resources, all volumes, no warnings.

  • Inspect the namespace-scoped MinIO backup in detail by running:
velero backup describe production-ns-minio --details
  • Look for these key fields in the output:
  • Phase: Completed confirms success.
  • The resource list should include Deployments, Services, PersistentVolumeClaims, CiliumNetworkPolicies, and BackupPolicy CRs.
  • The volumes section should show your postgres-pvc backed up via Kopia (file-system backup).
  • Check the S3 namespace backup the same way by running:
velero backup describe production-ns-s3 --details
  • Check the backup logs for any warnings or errors by running:
velero backup logs production-ns-minio | grep -i error
velero backup logs production-ns-s3 | grep -i error

Both commands should return no output if there were no errors during the backup process.

What makes a backup truly complete?

A complete backup for this project means: namespace resources captured (pods, services, network policies, CRs), PVC data captured via file-system backup, and CRD definitions captured in a separate cluster-scoped backup. Missing any layer means your restore will be incomplete.

The --details flag on velero backup describe shows the full resource list and volume backup details. This is your audit trail proving the backup is restorable.

Missing resources or volumes in the describe output?

  • If CiliumNetworkPolicies are missing from the namespace backup, ensure the CiliumNetworkPolicy CR exists in the production namespace (not just in kube-system).
  • If the PVC volume is not in the backup details, confirm the pod referencing it was Running at backup time. Velero only backs up volumes attached to Running pods.
  • If the CRD is not in your cluster-scoped backup, verify it was installed cluster-wide with kubectl get crd backuppolicies.dr.example.com.

Need help? Help me figure out why resources are missing from my Velero backup.

You now have a fully redundant backup set. Your production namespace, PVC data, CRDs, and network policies are safely stored in two independent locations. Next up, you will put these backups to the ultimate test by simulating a disaster and restoring everything from scratch.

Simulate Disaster and Restore

You now have four completed backups safely stored across two separate destinations. The real test of any DR strategy is not whether you can create backups. It is whether you can restore from them under pressure.

In this step, you will simulate a catastrophic namespace deletion and then perform a full restore from your local MinIO backups. You will validate that every component comes back intact: PostgreSQL data, Cilium policies, and custom CRDs.

In this step, get ready to:
  • Delete the entire production namespace and confirm all resources are gone.
  • Restore from MinIO backups in the correct order (CRDs first, then namespace resources).
  • Validate full recovery of data, network policies, and custom resources.
Simulate catastrophic failure

In a real disaster scenario, an entire namespace can vanish due to human error, a runaway automation, or a compromised credential. You will now intentionally destroy the production namespace to simulate exactly this.

  • Note the current time (you will use this to calculate your RTO later).
  • Delete the entire production namespace by running this command:
kubectl delete namespace production --wait=true

What does this command do?

The --wait=true flag blocks the terminal until Kubernetes has fully removed the namespace and all its contents.

This means pods, PVCs, services, secrets, CiliumNetworkPolicies, and custom resources within the namespace are all permanently deleted before the command returns.

  • Once the command completes, verify that all pods in the namespace are gone by running:
kubectl get pods -n production
  • You should see an error indicating the namespace does not exist, or a message showing no resources found.
  • Confirm the CiliumNetworkPolicy is also gone by running:
kubectl get ciliumnetworkpolicies -n production
  • Verify the custom resource instance is gone by running:
kubectl get backuppolicies.dr.example.com

You should see no resources for any of these commands. The namespace and everything in it has been wiped out.

Namespace deletion hanging?

If the delete command takes more than 2 minutes, a finalizer may be blocking it. Check for stuck resources with kubectl get all -n production.

Help me debug this issue

Why check the CRD separately?

CRD definitions are cluster-scoped resources, meaning deleting a namespace does not remove the CRD itself. However, if the CRD was only defined in the context of that namespace's workloads, its custom resource instances (CRs) are gone.

In some failure scenarios, the CRD definition can also be lost (e.g., accidental kubectl delete crd). Your cluster-scoped backup protects against this.

Restore from MinIO

With the disaster confirmed, you will now bring everything back from your local MinIO backups. The restore order matters: CRD definitions must exist before Velero can recreate the custom resource instances that depend on them.

  • Restore the cluster-scoped backup first (this recovers CRD definitions) by running:
velero restore create cluster-crds-restore --from-backup cluster-crds-minio

Why restore CRDs first?

Velero restores resources one at a time. If you restore the namespace backup first, Velero would try to create custom resource instances (like your main-backup-policy CR) before the CRD schema exists. That would cause those resources to fail with a validation error.

By restoring cluster-scoped CRDs first, the API server already knows about the backuppolicies.dr.example.com type when the namespace restore runs.

  • Wait for the CRD restore to complete, then check its status by running:
velero restore describe cluster-crds-restore
  • You should see Phase: Completed in the output with no errors listed.
  • Now restore the namespace-scoped backup (this brings back all production resources including PVC data) by running:
velero restore create production-ns-restore --from-backup production-ns-minio
  • This restore will take longer because it includes file-system backup data (your PostgreSQL PVC).
  • Wait about 30-60 seconds, then check the restore status by running:
velero restore describe production-ns-restore
  • You should see Phase: Completed with no errors.
  • Confirm all pods are back and running by checking:
kubectl get pods -n production
  • You should see the PostgreSQL pod in Running state with 1/1 ready.

Restore shows errors or PartiallyFailed?

Check the detailed restore logs with velero restore logs production-ns-restore. Common causes include the CRD restore not completing before the namespace restore started, or a node-agent pod not being ready.

If you see warnings about resources that already exist, those are non-blocking and can be safely ignored.

Help me diagnose restore failures

Validate full recovery

A restore is only successful if the application works exactly as it did before the disaster. You will now verify data integrity, network policy enforcement, and custom resource presence.

  • Query PostgreSQL to verify your seed data is intact by running:
kubectl exec -n production deploy/postgres -- psql -U postgres -d appdb -c "SELECT * FROM dr_test;"
  • You should see the same rows you inserted earlier in Step 2. This proves the file-system backup (via Kopia) correctly captured and restored your persistent volume data.
  • Verify the CiliumNetworkPolicy is restored and active by running:
kubectl get ciliumnetworkpolicies -n production
  • You should see postgres-ingress listed.
  • Verify the custom CRD and its resource instance are present by running:
kubectl get backuppolicies.dr.example.com
  • You should see main-backup-policy listed in the output.

What does this prove?

You have just validated three distinct categories of recovery:

  • Data integrity - PostgreSQL rows survived the disaster because Kopia captured the PVC at the filesystem level.
  • Network policy recovery - Cilium policies are CRDs themselves, and Velero treats them as namespace-scoped resources that restore alongside the workload.
  • CRD restoration - Your custom resource definition and its instance both came back because you ran a separate cluster-scoped backup.
  • Note the current time and calculate your RTO (Recovery Time Objective) by subtracting the time you started the delete from the time all validations passed.

Record your RTO

Your RTO on a local kind cluster is typically 1-3 minutes. In production, this number drives SLA commitments. Record it now so you can reference it in your validation below.

PostgreSQL data missing or query fails?

If the pod is running but the query returns an error, the PVC data may not have fully restored yet. Wait 30 seconds and try again.

If the dr_test table does not exist, check that your namespace restore included the PVC by running velero restore describe production-ns-restore and looking for volume restore entries.

Help me troubleshoot missing data after restore

You have proven that your dual-destination backup strategy works end to end. Your production namespace was completely destroyed and brought back to its exact pre-disaster state from local MinIO storage. Next up, you will extend this setup with automated scheduled backups in the Secret Mission.

Secret mission

Automated Scheduled Backups with Verification

You have proven you can back up and restore manually. In production, nobody runs backups by hand. Extend your DR setup with automated scheduled backups that run every 15 minutes to both storage destinations, plus a self-validating shell script that performs a mini disaster-recovery drill on demand.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running or delete them entirely. This project uses an AWS S3 bucket for offsite backup storage and a local kind cluster. The S3 storage cost is negligible (less than $0.01/month for the small amount of backup data stored).

Cost warning

Your AWS S3 bucket will accumulate a tiny ongoing storage charge (fractions of a cent per month) as long as it exists. If you enabled scheduled backups in the Secret Mission, new backup data will continue to land in S3 every 15 minutes until the kind cluster is deleted. Deleting the cluster stops new backups, but the existing data in S3 remains until you explicitly remove the bucket.

Resources you used:

  • AWS S3 bucket (velero-dr-project-{your-id}) storing Velero backup data.
  • kind cluster (includes MinIO, Velero server, Cilium CNI, PostgreSQL workload, and all in-cluster resources).
  • Velero CLI binary installed to your PATH.
  • Local project files (kind-config.yaml: cluster config, dr-verify.sh: DR verification script).

Keep everything running

No action needed. Choose this if you want to keep experimenting with restores, schedules, or backup configurations.

  • Your kind cluster continues running as Docker containers on your machine.
  • Scheduled backups (if configured) will keep creating new backup objects in both MinIO and S3 every 15 minutes.
  • S3 storage costs remain negligible for project-sized data.

Pause - I'll come back to this later

Stop the kind cluster's Docker containers to free up memory. Your S3 bucket remains with its existing data at negligible cost.

  • Stop the kind cluster's containers by running:
docker stop $(docker ps -q --filter "name=kind")
  • When you want to resume, start the containers again by running:
docker start $(docker ps -aq --filter "name=kind")

What happens when you pause?

Stopping the Docker containers halts the kind cluster without destroying it. Your workloads, Velero configuration, and backup history all persist on disk. Scheduled backups will not run while the cluster is stopped. When you restart, everything resumes from where you left off.

Delete - I don't want to use this again

Remove all project resources completely. This frees up disk space and stops any S3 charges.

  • Delete the kind cluster (removes all in-cluster resources including MinIO, Velero server, Cilium, and workloads) by running:
kind delete cluster
  • Delete the AWS S3 bucket and all its contents by running:
aws s3 rb s3://[[S3_BUCKET="velero-dr-project-your-id"]] --force
  • Remove the Velero CLI binary by running:
sudo rm /usr/local/bin/velero
  • Remove local project files by running:
rm kind-config.yaml dr-verify.sh

What about the Cilium CLI?

If you installed the Cilium CLI during this project and no longer need it, you can remove it the same way:

sudo rm /usr/local/bin/cilium

If you use Cilium for other projects, leave it installed.

Nice Work!

Nice Work!

You just built a production-grade disaster recovery system for Kubernetes using Velero with a dual-destination backup strategy. You can now confidently destroy a namespace and restore it with all data, policies, and custom resources intact.

You've learned how to:

  • Deploy a stateful application stack on a kind cluster with Cilium CNI, PostgreSQL with persistent storage, CiliumNetworkPolicies, and custom CRDs.
  • Configure Velero with dual BackupStorageLocations (in-cluster MinIO + AWS S3) and execute full-scope backups covering namespace resources, PVC data via Kopia file-system backup, cluster-scoped CRDs, and CNI policies.
  • Simulate a catastrophic namespace deletion and perform a validated restore proving PostgreSQL data integrity, CiliumNetworkPolicy enforcement, and CRD recovery.
  • (Secret Mission) Automate the backup lifecycle with scheduled backups, retention policies, and a self-validating DR drill script that proves your disaster recovery pipeline works end-to-end without manual intervention.

Ready to quiz yourself?