Deploy Rook-Ceph Storage on EKS

Deploy a distributed Ceph storage cluster on Amazon EKS using the Rook operator.

Introduction

30 Second Summary

Modern applications need storage that survives server failures, scales on demand, and doesn't live on a single machine. Distributed storage systems solve this by spreading data across multiple nodes, but setting them up manually is notoriously complex.

In this project, you will deploy a full Ceph distributed storage cluster on Amazon EKS using the Rook operator. You'll learn how Kubernetes operators manage complex infrastructure through Custom Resource Definitions and reconciliation loops.

What You'll Build

A production-style distributed storage cluster running on Amazon EKS, managed entirely through Kubernetes-native custom resources and the Rook operator.

By the end of this project, you'll have:

  • A 3-node EKS cluster with the EBS CSI driver for dynamic volume provisioning.
  • A Rook-Ceph operator deployment that automatically manages the lifecycle of a complete Ceph storage cluster (monitors, OSDs, and a manager daemon).
  • A CephBlockPool and StorageClass for dynamically provisioning Ceph block storage to workloads.
  • Secret Mission: Set up a CephFS shared filesystem for multi-pod ReadWriteMany access.

Are there any prerequisites?

You need an AWS account with the AWS CLI already configured (credentials and default region set). You also need Docker installed on your machine.

Create a 3-Node EKS Cluster with EBS CSI Driver

Your tools are installed and ready to go. Now it's time to create the Kubernetes cluster that will host your distributed Ceph storage system.

In this step, you'll use eksctl to provision a 3-node Amazon EKS cluster with the EBS CSI driver add-on. The EBS CSI driver allows Kubernetes to dynamically create and attach EBS volumes. Rook-Ceph will use these volumes as the underlying storage for its OSDs.

In this step, get ready to:
  • Create an EKS cluster configuration file and launch the cluster.
  • Verify all nodes are ready and the EBS CSI driver is running.
  • Create a StorageClass for the EBS CSI provisioner.
Create the cluster configuration and launch the cluster

An eksctl cluster config file defines everything about your cluster in a single YAML document. This includes the node group size, instance type, IAM settings, and add-ons.

  • Create a file called cluster.yaml by running:
cat <<'EOF' > cluster.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: rook-ceph-cluster
  region: us-east-1

iam:
  withOIDC: true

managedNodeGroups:
  - name: ceph-nodes
    instanceType: t3.xlarge
    desiredCapacity: 3
    volumeSize: 30
    iam:
      withAddonPolicies:
        ebs: true

addons:
  - name: aws-ebs-csi-driver
    attachPolicyARNs:
      - arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicyV2
EOF

What does this configuration do?

  • metadata names the cluster rook-ceph-cluster and places it in us-east-1.
  • withOIDC: true enables IAM Roles for Service Accounts (IRSA), which the EBS CSI driver needs to access AWS APIs securely.
  • managedNodeGroups creates 3 t3.xlarge instances (4 vCPUs, 16 GB RAM each). Ceph MON and OSD daemons need at least this much memory to run reliably.
  • addons installs the aws-ebs-csi-driver with the AmazonEBSCSIDriverPolicyV2 IAM policy attached via IRSA.
  • Launch the cluster by running:
eksctl create cluster -f cluster.yaml

This takes approximately 15 minutes

Cluster creation provisions a VPC, subnets, an EKS control plane, and 3 EC2 instances. This is a good time to learn about the Ceph architecture you'll deploy in the next steps.

What is Ceph?

Ceph is a distributed storage system that spreads data across multiple machines for reliability and performance. It has three core daemon types:

  • MONs (Monitors) maintain consensus about the cluster map. Your cluster will run 3 MONs to form a quorum.
  • OSDs (Object Storage Daemons) store the actual data. Each OSD manages one storage device. Your cluster will have 3 OSDs, each backed by an EBS volume.
  • MGR (Manager) provides monitoring and orchestration features like the dashboard and metrics.

Cluster creation failed?

  • Make sure your AWS CLI is configured with credentials that have permissions to create EKS clusters, EC2 instances, VPCs, and IAM roles.
  • Check that your AWS account has not hit service limits for EC2 instances or VPCs in us-east-1.
  • If you see an OIDC-related error, ensure your eksctl version supports the withOIDC field.

Why is my eksctl cluster creation failing?

Verify the cluster is ready

Once eksctl finishes, it automatically configures your kubectl context to point at the new cluster.

  • Confirm all 3 nodes are in a Ready state by running:
kubectl get nodes

You should see 3 nodes listed, all with a STATUS of Ready.

  • Verify the EBS CSI driver controller pods are running by running:
kubectl get pods -n kube-system -l app=ebs-csi-controller

You should see 2 controller pods in a Running state.

Why 2 controller pods?

The EBS CSI driver deploys 2 controller pod replicas by default for high availability. If one controller pod fails, the other continues handling volume operations without interruption.

Nodes not showing Ready?

  • Wait a minute and retry. Nodes can take a short time to transition to Ready after the cluster reports as created.
  • If no nodes appear at all, check that eksctl completed successfully. Look for errors in the terminal output from the previous step.
  • Run kubectl config current-context to confirm your context points to the new cluster.

Why are my EKS nodes not showing Ready?

Create the gp2-csi StorageClass

EKS clusters include a default gp2 StorageClass that uses the legacy in-tree provisioner. Rook-Ceph needs a StorageClass that uses the CSI driver instead. You'll create one called gp2-csi.

  • Create a file called gp2-csi-storageclass.yaml by running:
cat <<'EOF' > gp2-csi-storageclass.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp2-csi
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  fsType: ext4
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
reclaimPolicy: Delete
EOF

What does this StorageClass do?

  • provisioner: ebs.csi.aws.com tells Kubernetes to use the EBS CSI driver for volume operations.
  • type: gp3 provisions gp3 EBS volumes, the latest generation general-purpose SSD.
  • volumeBindingMode: WaitForFirstConsumer delays volume creation until a pod actually needs it. This ensures the volume is created in the same availability zone as the pod.
  • allowVolumeExpansion: true lets you resize volumes later without recreating them.
  • Apply the StorageClass to your cluster by running:
kubectl apply -f gp2-csi-storageclass.yaml
  • Verify the StorageClass was created by running:
kubectl get storageclass gp2-csi

You should see the gp2-csi StorageClass listed with ebs.csi.aws.com as its provisioner.

StorageClass not appearing?

  • Check for YAML syntax errors in gp2-csi-storageclass.yaml. YAML is sensitive to indentation.
  • Make sure you ran kubectl apply -f gp2-csi-storageclass.yaml and saw storageclass.storage.k8s.io/gp2-csi created in the output.

Why isn't my StorageClass showing up in kubectl?

✔️ Awesome, I've got everything!

Great. Double check you've saved both files and that the StorageClass is visible in your cluster.

ⓧ I'd like to double check the full code

Here is the complete cluster.yaml:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: rook-ceph-cluster
  region: us-east-1

iam:
  withOIDC: true

managedNodeGroups:
  - name: ceph-nodes
    instanceType: t3.xlarge
    desiredCapacity: 3
    volumeSize: 30
    iam:
      withAddonPolicies:
        ebs: true

addons:
  - name: aws-ebs-csi-driver
    attachPolicyARNs:
      - arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicyV2

Here is the complete gp2-csi-storageclass.yaml:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp2-csi
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  fsType: ext4
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
reclaimPolicy: Delete

Your EKS cluster is running with 3 healthy nodes and the EBS CSI driver ready to provision volumes. Next up, you'll deploy the Rook operator that will use this infrastructure to create and manage a full Ceph storage cluster.

Deploy the Rook Operator

Your EKS cluster is running with 3 nodes and the EBS CSI driver is ready to provision volumes. Now you need something to orchestrate the Ceph storage cluster on top of this infrastructure.

The Rook operator is a Kubernetes operator that watches for Ceph-related Custom Resources and automatically reconciles them into running Ceph daemons. In this step, you will install the Rook operator using Helm and explore the resources it creates.

In this step, get ready to:
  • Install the Rook-Ceph operator via Helm into the rook-ceph namespace.
  • Examine the CRDs, deployment, and RBAC resources the operator creates.
Install the Rook-Ceph operator with Helm

Helm charts bundle all the Kubernetes manifests (Deployments, RBAC rules, CRDs) that an operator needs into a single installable package. The Rook project publishes an official chart that sets up everything the operator requires.

  • Add the Rook Helm repository by running this command:
helm repo add rook-release https://charts.rook.io/release
  • Update your local Helm repository index by running:
helm repo update
  • Install the Rook-Ceph operator chart into a new rook-ceph namespace by running:
helm install --create-namespace --namespace rook-ceph rook-ceph rook-release/rook-ceph

What does this command do?

This Helm command does several things at once:

  • --create-namespace --namespace rook-ceph creates the rook-ceph namespace if it does not already exist, then installs into it.
  • rook-ceph (the third argument) is the Helm release name you are giving this installation.
  • rook-release/rook-ceph points to the chart from the repository you just added.
  • Wait about 30 seconds for the operator pod to start, then verify it is running:
kubectl -n rook-ceph get pod

You should see a pod named rook-ceph-operator-<hash> with status Running and 1/1 containers ready.

Operator pod not showing as Running?

  • If the pod shows ContainerCreating, wait another 30 seconds and run the command again. Image pulls can take time on first deploy.
  • If you see CrashLoopBackOff, check the logs with kubectl -n rook-ceph logs deploy/rook-ceph-operator for specific error messages.
  • Make sure your kubectl context is set to your EKS cluster. Run kubectl config current-context to confirm.

Help me troubleshoot my Rook operator pod not reaching Running state on EKS.

Examine what the operator created

The Rook operator does not just deploy a pod. It registers new resource types (CRDs) with the Kubernetes API, sets up RBAC permissions, and starts a controller that watches for those custom resources.

  • List the Ceph-related CRDs the operator installed by running:
kubectl get crd | grep ceph

You should see several CRDs including cephclusters.ceph.rook.io, cephblockpools.ceph.rook.io, and cephfilesystems.ceph.rook.io. These are the APIs you will use to tell the operator what to build.

  • View the operator deployment details by running:
kubectl -n rook-ceph get deployment
  • Check the RBAC resources (ClusterRoles and ServiceAccounts) the operator created by running:
kubectl get clusterrole | grep rook

What is the operator pattern?

This is the operator pattern in action. The operator installed CRDs that extend the Kubernetes API with new resource types like CephCluster and CephBlockPool.

When you create one of these custom resources in the next step, the operator's controller detects the change and reconciles it. That means it automatically creates the pods, services, and configuration needed to make the desired state a reality. You describe what you want. The operator figures out how to build it.

Not seeing any CRDs?

  • Make sure the Helm install completed successfully. Run helm list -n rook-ceph to confirm the release exists with status deployed.
  • CRDs are cluster-scoped. If you have multiple contexts, verify you are querying the correct cluster.

Help me figure out why kubectl get crd shows no Ceph CRDs after installing the Rook Helm chart.

The Rook operator is deployed and watching for Ceph custom resources. Next up, you will create a CephCluster resource that tells the operator to spin up a full Ceph storage cluster across your 3 EKS nodes.

Deploy the Ceph Cluster and Verify Health

With the Rook operator watching for CephCluster custom resources, you can now define your desired storage cluster state in a YAML file. The operator reads this file and reconciles reality to match it. It will provision EBS volumes, launch MON daemons, spin up OSDs, and start the MGR.

In this step, you will create the CephCluster custom resource, watch the operator bring the cluster to life, then verify the cluster is healthy using the Ceph toolbox.

In this step, get ready to:
  • Create and apply the CephCluster custom resource with 3 MONs and 3 OSDs.
  • Deploy the Rook toolbox and verify cluster health reaches HEALTH_OK.
  • Explore Ceph internals with the CRUSH map and storage capacity commands.
Create and apply the CephCluster custom resource

A CephCluster CR tells the Rook operator exactly what storage cluster you want. You declare the Ceph version, how many monitors and OSDs to run, and which StorageClass to use for their backing volumes.

  • Create a file called cephcluster.yaml in your working directory with the following content:
apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph
  namespace: rook-ceph
spec:
  cephVersion:
    image: quay.io/ceph/ceph:v20.2.1
  dataDirHostPath: /var/lib/rook
  mon:
    count: 3
    allowMultiplePerNode: false
    volumeClaimTemplate:
      spec:
        storageClassName: gp2-csi
        resources:
          requests:
            storage: 10Gi
  dashboard:
    enabled: true
  storage:
    storageClassDeviceSets:
      - name: set1
        count: 3
        portable: false
        encrypted: false
        volumeClaimTemplates:
          - metadata:
              name: data
            spec:
              resources:
                requests:
                  storage: 10Gi
              storageClassName: gp2-csi
              volumeMode: Block
              accessModes:
                - ReadWriteOnce
    onlyApplyOSDPlacement: false

What does this configuration do?

The mon section declares 3 monitor daemons, each backed by a 10Gi EBS volume provisioned through the gp2-csi StorageClass. MONs maintain the cluster map and consensus.

The storageClassDeviceSets section tells the operator to create 3 OSDs (Object Storage Daemons), each with a 10Gi block-mode EBS volume. OSDs handle the actual data storage and replication.

Setting volumeMode: Block means the EBS volume is presented as a raw block device to Ceph, which is what Ceph expects for its BlueStore backend.

  • Apply the CephCluster CR to your cluster by running:
kubectl apply -f cephcluster.yaml
  • Watch the operator reconcile the desired state into running pods by running:
kubectl -n rook-ceph get pods -w

Over the next 5-10 minutes, you will see MON pods, MGR pods, and OSD pods appear and transition to Running status. Press Ctrl+C to stop watching once you see all pods running.

What is the operator doing right now?

The Rook operator detected the CephCluster CR you just applied. It is now creating PVCs for the MONs and OSDs, waiting for EBS volumes to provision via the CSI driver, then launching the Ceph daemons in the correct order. Monitors start first to establish quorum, then the manager, then the OSDs join the cluster.

This is the operator pattern in action. You declared your desired state, and the controller is reconciling reality to match it. If a pod fails, the operator will restart it automatically.

Pods stuck in Pending?

If MON or OSD pods stay in Pending for more than 5 minutes, check that your gp2-csi StorageClass exists and the EBS CSI driver pods are running. Run kubectl get sc and kubectl get pods -n kube-system -l app=ebs-csi-controller to verify.

Also confirm your nodes have enough resources. Each t3.xlarge has 16GB RAM. With 3 OSDs and MONs on 3 nodes, you need about 4-5GB per node for Ceph daemons.

help me debug pods stuck in Pending in my Rook-Ceph deployment

Deploy the toolbox and verify cluster health

The Rook toolbox is a pod that contains all the Ceph CLI tools. You will use it to run ceph status and confirm the cluster reached HEALTH_OK.

  • Deploy the Rook toolbox from the official examples by running:
kubectl create -f https://raw.githubusercontent.com/rook/rook/master/deploy/examples/toolbox.yaml
  • Wait for the toolbox pod to reach Running state by running:
kubectl -n rook-ceph rollout status deploy/rook-ceph-tools

You should see a message indicating the deployment was successfully rolled out.

  • Connect to the toolbox pod by running:
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- bash
  • Inside the toolbox, check the cluster health by running:
ceph status

What should I see?

You should see output showing health: HEALTH_OK with 3 MONs in quorum, an active MGR, and 3 OSDs up and in. If you see HEALTH_WARN, wait a few more minutes for the cluster to stabilize. The cluster is operational as long as all monitors are in quorum and OSDs are up.

Seeing HEALTH_WARN or less than 3 OSDs?

If the cluster shows fewer than 3 OSDs up, check that all OSD pods are Running with kubectl -n rook-ceph get pods -l app=rook-ceph-osd. OSD prepare jobs must complete before the OSD daemons start.

If you see HEALTH_WARN about pools not having enough replicas, wait another 2-3 minutes. The cluster needs time to peer all placement groups after the OSDs come online.

help me troubleshoot my Ceph cluster health status

Explore the Ceph cluster internals

Now that the cluster is healthy, you can explore its internal structure. The CRUSH map determines how Ceph distributes data across OSDs, and ceph df shows how much raw storage is available.

  • Still inside the toolbox, view the CRUSH map topology by running:
ceph osd tree

What does the OSD tree show?

The ceph osd tree command displays the CRUSH map hierarchy. You will see a root node, with host-level buckets underneath it, and individual OSDs under each host. This topology tells Ceph how to replicate data across failure domains. With failureDomain: host, each replica goes to a different node.

  • View the cluster's raw storage capacity by running:
ceph df

You should see approximately 30Gi of total raw storage (3 OSDs x 10Gi each), with most of it available since no data has been written yet.

  • Exit the toolbox shell by running:
exit
  • Finally, examine the CephCluster CR status from outside the toolbox by running:
kubectl get cephcluster -n rook-ceph -o yaml

Why check the CR status?

The CephCluster CR's status field is updated by the operator to reflect the actual cluster state. This is the Kubernetes-native way to observe the health of operator-managed resources. You will see the phase, Ceph health status, and details about MONs, MGR, and OSDs all in one place.

✔️ Awesome, I've got everything!

Great. Double check you have saved cephcluster.yaml and that your cluster shows HEALTH_OK from both the toolbox and the CR status output.

ⓧ I'd like to double check the full code

Here is the complete cephcluster.yaml for comparison:

apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph
  namespace: rook-ceph
spec:
  cephVersion:
    image: quay.io/ceph/ceph:v20.2.1
  dataDirHostPath: /var/lib/rook
  mon:
    count: 3
    allowMultiplePerNode: false
    volumeClaimTemplate:
      spec:
        storageClassName: gp2-csi
        resources:
          requests:
            storage: 10Gi
  dashboard:
    enabled: true
  storage:
    storageClassDeviceSets:
      - name: set1
        count: 3
        portable: false
        encrypted: false
        volumeClaimTemplates:
          - metadata:
              name: data
            spec:
              resources:
                requests:
                  storage: 10Gi
              storageClassName: gp2-csi
              volumeMode: Block
              accessModes:
                - ReadWriteOnce
    onlyApplyOSDPlacement: false

Your Ceph cluster is healthy and operational. Next up, you will create a block storage pool and StorageClass, then deploy a workload that reads and writes data to Ceph-backed persistent storage.

Provision Block Storage and Deploy a Workload

In the previous step you deployed a healthy Ceph cluster and verified it from the inside using the toolbox. Your cluster has storage capacity, but applications cannot use it yet.

To bridge Ceph and Kubernetes workloads, you need a CephBlockPool (defining how data is replicated) and a StorageClass (exposing that pool to Kubernetes via the CSI driver). In this step, you will create both, then deploy a pod that writes and reads from Ceph-backed block storage.

In this step, get ready to:
  • Create a CephBlockPool and StorageClass for Ceph block storage.
  • Deploy a test workload that writes data to a Ceph-backed PersistentVolumeClaim.
  • Verify data persistence and observe the full storage stack from PVC to RBD image.
Create the CephBlockPool and StorageClass

A CephBlockPool tells Ceph how to replicate data across your OSDs. The StorageClass wraps that pool so any Kubernetes workload can dynamically request block volumes without knowing Ceph internals.

  • Create a file named cephblockpool.yaml with the following content:
apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
  name: replicapool
  namespace: rook-ceph
spec:
  failureDomain: host
  replicated:
    size: 3
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rook-ceph-block
provisioner: rook-ceph.rbd.csi.ceph.com
parameters:
  clusterID: rook-ceph
  pool: replicapool
  imageFormat: "2"
  imageFeatures: layering
  csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
  csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph
  csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
  csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
  csi.storage.k8s.io/fstype: ext4
reclaimPolicy: Delete
allowVolumeExpansion: true

What does this file define?

The first resource is a CephBlockPool named replicapool. It sets failureDomain: host so Ceph distributes replicas across different worker nodes. With replicated.size: 3, every block of data is stored three times for fault tolerance.

The second resource is a StorageClass named rook-ceph-block. Its provisioner rook-ceph.rbd.csi.ceph.com is the Rook CSI driver for RBD (RADOS Block Device) volumes. The parameters point the driver at the correct pool, cluster, and authentication secrets that the Rook operator created automatically.

  • Apply both resources by running:
kubectl apply -f cephblockpool.yaml
  • Verify the pool and StorageClass exist by running:
kubectl get cephblockpool -n rook-ceph
  • You should see replicapool listed. Now check the StorageClass by running:
kubectl get storageclass rook-ceph-block

You should see rook-ceph-block with provisioner rook-ceph.rbd.csi.ceph.com listed.

StorageClass not appearing?

Make sure you included the --- YAML document separator between the CephBlockPool and StorageClass in your file. Without it, kubectl treats both as a single malformed resource.

Also verify the Rook operator pod is still running in the rook-ceph namespace. The operator must be healthy to reconcile the CephBlockPool.

Ask for help: Why isn't my rook-ceph-block StorageClass appearing after applying cephblockpool.yaml?

✔️ Awesome, I've got everything!

Great. Make sure you have saved cephblockpool.yaml and confirmed both resources exist before moving on.

ⓧ I'd like to double check the full code

Here is the complete cephblockpool.yaml for reference:

apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
  name: replicapool
  namespace: rook-ceph
spec:
  failureDomain: host
  replicated:
    size: 3
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rook-ceph-block
provisioner: rook-ceph.rbd.csi.ceph.com
parameters:
  clusterID: rook-ceph
  pool: replicapool
  imageFormat: "2"
  imageFeatures: layering
  csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
  csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph
  csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
  csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
  csi.storage.k8s.io/fstype: ext4
reclaimPolicy: Delete
allowVolumeExpansion: true
Deploy a test workload and verify data persistence

With the StorageClass in place, any pod can request Ceph block storage through a standard PersistentVolumeClaim. The CSI driver handles the rest: creating an RBD image in Ceph, attaching it to the node, formatting it, and mounting it into the container.

  • Create a file named test-workload.yaml with the following content:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ceph-block-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  storageClassName: rook-ceph-block
---
apiVersion: v1
kind: Pod
metadata:
  name: ceph-test-pod
spec:
  containers:
    - name: test
      image: busybox
      command: ["/bin/sh", "-c"]
      args:
        - |
          echo "Hello from Ceph storage!" > /mnt/ceph/testfile.txt
          cat /mnt/ceph/testfile.txt
          sleep 3600
      volumeMounts:
        - name: ceph-vol
          mountPath: /mnt/ceph
  volumes:
    - name: ceph-vol
      persistentVolumeClaim:
        claimName: ceph-block-pvc

What does this workload do?

The PVC named ceph-block-pvc requests 5Gi of storage from the rook-ceph-block StorageClass. Kubernetes asks the Rook CSI driver to create a 5Gi RBD image inside replicapool.

The pod mounts that PVC at /mnt/ceph, writes a test file, reads it back, then sleeps so you can inspect it. The busybox image is a lightweight container with basic shell utilities.

  • Deploy the PVC and pod by running:
kubectl apply -f test-workload.yaml
  • Wait for the pod to reach Running state by running:
kubectl get pod ceph-test-pod -w

Once you see STATUS change to Running, press Ctrl+C to stop watching.

  • Confirm the file was written by checking the pod logs:
kubectl logs ceph-test-pod

You should see Hello from Ceph storage! printed in the output.

Pod stuck in Pending or ContainerCreating?

Check the PVC status with kubectl get pvc ceph-block-pvc. If it shows Pending, the CSI driver may not be able to provision the volume.

Run kubectl describe pvc ceph-block-pvc and look at the Events section for provisioning errors. Common causes include the CephBlockPool not being ready or the CSI driver pods not running.

Ask for help: My ceph-test-pod is stuck in Pending after applying test-workload.yaml. How do I debug PVC provisioning?

Now prove that the data survives a pod restart. This is the key value of persistent storage.

  • Delete the pod (leaving the PVC intact) by running:
kubectl delete pod ceph-test-pod
  • Recreate the pod by re-applying the same manifest:
kubectl apply -f test-workload.yaml
  • Wait for the pod to reach Running state again, then read the file directly:
kubectl exec ceph-test-pod -- cat /mnt/ceph/testfile.txt

You should see Hello from Ceph storage! again. The data survived because it lives in Ceph, not inside the container filesystem.

Why does the data persist?

When you deleted the pod, Kubernetes detached the RBD volume from the node. The PVC and its underlying RBD image in Ceph remained intact. Recreating the pod re-attached the same volume, so the file was still there.

This is the same mechanism production databases use to survive pod restarts and node failures.

✔️ Awesome, I've got everything!

Great. Make sure you have saved test-workload.yaml and confirmed data persistence before moving on.

ⓧ I'd like to double check the full code

Here is the complete test-workload.yaml for reference:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ceph-block-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  storageClassName: rook-ceph-block
---
apiVersion: v1
kind: Pod
metadata:
  name: ceph-test-pod
spec:
  containers:
    - name: test
      image: busybox
      command: ["/bin/sh", "-c"]
      args:
        - |
          echo "Hello from Ceph storage!" > /mnt/ceph/testfile.txt
          cat /mnt/ceph/testfile.txt
          sleep 3600
      volumeMounts:
        - name: ceph-vol
          mountPath: /mnt/ceph
  volumes:
    - name: ceph-vol
      persistentVolumeClaim:
        claimName: ceph-block-pvc
Observe the full storage stack

You have proven that data persists. Now trace the entire path from Kubernetes PVC down through the Ceph RBD image to understand every layer involved.

  • View the PVC and its bound PersistentVolume by running:
kubectl get pvc ceph-block-pvc

You should see STATUS as Bound and a VOLUME column showing the dynamically created PV name.

  • Inspect the PersistentVolume to see its CSI details by running:
kubectl get pv $(kubectl get pvc ceph-block-pvc -o jsonpath='{.spec.volumeName}') -o yaml

Look for the csi section in the output. It shows the RBD image name, pool, and cluster ID that the CSI driver uses to mount the volume.

  • Now see the actual RBD image inside Ceph. Switch to the Rook toolbox from earlier by running:
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- rbd ls replicapool

You should see an RBD image name listed (a CSI-prefixed string). This is the block device that backs your PVC.

  • Finally, describe the pod to see the volume mount details by running:
kubectl describe pod ceph-test-pod

In the output, find the Volumes and Mounts sections. You can trace the chain: PVC references the StorageClass, the CSI driver provisions an RBD image, and the kubelet mounts it at /mnt/ceph inside the container.

What is the full storage path?

  • Pod requests a volume via a PVC.
  • Kubernetes matches the PVC to the rook-ceph-block StorageClass.
  • The Rook CSI driver creates an RBD image in replicapool and provisions a PV.
  • Ceph replicates that image's data across 3 OSDs on 3 different hosts.
  • The kubelet on the scheduled node maps the RBD device and mounts it into the container.

Secret mission

Set Up a CephFS Shared Filesystem

So far you've provisioned block storage, which only one pod can mount at a time. What if multiple pods need to read and write the same files simultaneously? In this secret mission, you'll set up CephFS for true shared filesystem access across pods.

Clean Up Your Resources

Clean Up Your Resources

Decide whether to keep your resources running, pause them to come back later, or delete them entirely.

This project uses paid AWS resources that cost approximately $3-4 per hour while running.

Cost warning

Your EKS cluster costs approximately $3-4 per hour while running. This includes the EKS control plane ($0.10/hr), three t3.xlarge EC2 instances (~$0.50/hr total), and EBS volumes for Ceph storage (~$0.10/hr).

If you are done experimenting, delete the cluster immediately to avoid unexpected charges.

Resources you used:

  • EKS cluster rook-ceph-cluster (3 t3.xlarge worker nodes, EKS control plane).
  • EBS volumes (6x 10Gi for Ceph MONs and OSDs, plus node root volumes).
  • Rook-Ceph operator Helm release and all Ceph resources (CephCluster, CephBlockPool, CephFilesystem).
  • Local YAML files (cluster.yaml, gp2-csi-storageclass.yaml, cephcluster.yaml, cephblockpool.yaml, test-workload.yaml, cephfilesystem.yaml, shared-workload.yaml).
  • Installed CLI tools (eksctl, kubectl, helm).

Keep everything running

No action needed. Choose this if you are still actively experimenting with Ceph storage or want to try additional configurations.

  • Your cluster continues running and costs approximately $3-4 per hour.
  • All Ceph data, StorageClasses, and workloads remain intact.
  • The CLI tools (eksctl, kubectl, helm) remain installed on your machine.

Pause - I'll come back to this later

Pausing is not possible with this project. Scaling the EKS nodegroup to 0 nodes would break the Ceph cluster, because Ceph requires its MON and OSD daemons to maintain quorum.

  • If you want to stop costs, you must delete the cluster entirely and recreate it later.
  • Your local YAML files will still be on your machine, so recreating the cluster is straightforward. Run the same eksctl create cluster and kubectl apply commands from the project steps.

Delete - I don't want to use this again

Remove all project resources and stop all AWS charges. This is recommended if you are done with the project.

  • Delete the EKS cluster and all associated AWS resources by running:
eksctl delete cluster -f cluster.yaml --disable-nodegroup-eviction

What does this command do?

This deletes the EKS cluster, all worker nodes, the EBS CSI driver, and the associated CloudFormation stacks. The --disable-nodegroup-eviction flag bypasses Pod Disruption Budget policies that the Ceph operator sets, which would otherwise block node draining.

  • Wait for the command to complete. This takes several minutes.
  • Verify in the AWS Console that no orphaned resources remain. Open the CloudFormation service and confirm no stacks with rook-ceph-cluster in the name are left behind.
  • Open the EC2 service, select Volumes in the left sidebar, and confirm no orphaned EBS volumes remain in the us-east-1 region.
  • Delete the local project files by running:
rm cluster.yaml gp2-csi-storageclass.yaml cephcluster.yaml cephblockpool.yaml test-workload.yaml cephfilesystem.yaml shared-workload.yaml
  • Remove the installed CLI tools by running:
rm /usr/local/bin/eksctl
rm /usr/local/bin/kubectl
rm /usr/local/bin/helm

Nice Work!

Nice Work!

Nice work! You just deployed and operated a production-style distributed storage cluster on Amazon EKS using the Rook operator pattern.

You've learned how to:

  • Deploy the Rook-Ceph operator on a 3-node EKS cluster and use CRDs and reconciliation loops to manage a complete Ceph storage cluster (MONs, OSDs, MGR) backed by dynamically provisioned EBS volumes.
  • Explore Ceph internals including the CRUSH map, OSD tree, and cluster health monitoring to understand distributed storage topology.
  • Provision Ceph block storage via a StorageClass and consume it with a workload, proving end-to-end dynamic storage provisioning on Kubernetes.
  • Secret Mission: Set up a CephFS shared filesystem with ReadWriteMany multi-pod access.

Ready to quiz yourself?