<p>I'm trying to setup my Kubernetescluster with a Ceph Cluster using a storageClass, so that with each PVC a new PV is created automatically inside the ceph cluster.</p>

<p>But it doesn't work. I've tried a lot and read a lot of documentation and tutorials and can't figure out, what went wrong.</p>

<p>I've created 2 secrets, for the ceph admin user and an other user <code>kube</code>, which I created with this command to grant access to a ceph osd pool.</p>

<p>Creating the pool:
<code>sudo ceph osd pool create kube 128</code></p>

<p>Creating the user:
<code>sudo ceph auth get-or-create client.kube mon 'allow r' \
osd 'allow class-read object_prefix rbd_children, allow rwx pool=kube' \
-o /etc/ceph/ceph.client.kube.keyring</code></p>

<p>After that I exported both the keys and converted them to Base64 with:
<code>sudo ceph auth get-key client.admin | base64</code> and <code>sudo ceph auth get-key client.kube | base64</code>
I used those values inside my secret.yaml to create kubernetes secrets.</p>

<pre><code>apiVersion: v1
kind: Secret
type: "kubernetes.io/rbd"
metadata:
  name: ceph-secret
data:
   key: QVFCb3NxMVpiMVBITkJBQU5ucEEwOEZvM1JlWHBCNytvRmxIZmc9PQo=
</code></pre>

<p>And another one named ceph-user-secret.</p>

<p>Then I created a storage class to use the ceph cluster</p>

<pre><code>apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: standard
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: kubernetes.io/rbd

parameters:
  monitors: publicIpofCephMon1:6789,publicIpofCephMon2:6789
  adminId: admin
  adminSecretName: ceph-secret
  pool: kube

  userId: kube
  userSecretName: ceph-kube-secret

  fsType: ext4
  imageFormat: "2"
  imageFeatures: "layering"
</code></pre>

<p>To test my setup I created a PVC</p>

<pre><code>apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-eng
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
</code></pre>

<p>But it gets stuck in the pending state:</p>

<pre><code>#kubectl get pvc
NAME      STATUS    VOLUME    CAPACITY   ACCESSMODES   STORAGECLASS   AGE
pvc-eng   Pending                                      standard       25m
</code></pre>

<p>Also, no images are created inside the ceph kube pool.
Do you have any recommendations how to debug this problem?</p>

<p>I tried to install the ceph-common ubuntu package on all kubernetes nodes. I switched the kube-controller-manager docker image with an image provided by AT&amp;T which includes the ceph-common package.</p>

<p><a href="https://github.com/att-comdev/dockerfiles/tree/master/kube-controller-manager" rel="nofollow noreferrer">https://github.com/att-comdev/dockerfiles/tree/master/kube-controller-manager</a> </p>

<p>Network is fine, I can access my ceph cluster from inside a pod and from every kubernetes host.</p>

<p>I would be glad if anyone has any ideas!</p>

<p>You must use <code>annotation: ReadWriteOnce</code>.
As you can see <a href="https://kubernetes.io/docs/concepts/storage/persistent-volumes/" rel="nofollow noreferrer">https://kubernetes.io/docs/concepts/storage/persistent-volumes/</a> (Persistent Volumes section) RBD devices does not support ReadWriteMany mode. Choose different volume plugin (CephFS, for example) if you need read and write data from PV by several pods.</p>
