title: CephFS Unable to attach or mount volumes: unmounted volumes=[image-store]
body: <p>I'm having trouble getting my Kube-registry up and running on cephfs. I'm using rook to set this cluster up. As you can see, I'm having trouble attaching the volume. Any idea what would be causing this issue? any help is appreciated.</p>
  <p><strong>kube-registry.yaml</strong></p>
  <pre><code>    apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        name: cephfs-pvc
        namespace: kube-system
      spec:
        accessModes:
        - ReadWriteMany
        resources:
          requests:
            storage: 1Gi
        storageClassName: rook-cephfs
      ---
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: kube-registry
        namespace: kube-system
        labels:
          k8s-app: kube-registry
          kubernetes.io/cluster-service: &quot;true&quot;
      spec:
        replicas: 3
        selector:
          matchLabels:
            k8s-app: kube-registry
        template:
          metadata:
            labels:
              k8s-app: kube-registry
              kubernetes.io/cluster-service: &quot;true&quot;
          spec:
            containers:
            - name: registry
              image: registry:2
              imagePullPolicy: Always
              resources:
                limits:
                  cpu: 100m
                  memory: 100Mi
              env:
              # Configuration reference: https://docs.docker.com/registry/configuration/
              - name: REGISTRY_HTTP_ADDR
                value: :5000
              - name: REGISTRY_HTTP_SECRET
                value: &quot;Ple4seCh4ngeThisN0tAVerySecretV4lue&quot;
              - name: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY
                value: /var/lib/registry
              volumeMounts:
              - name: image-store
                mountPath: /var/lib/registry
              ports:
              - containerPort: 5000
                name: registry
                protocol: TCP
              livenessProbe:
                httpGet:
                  path: /
                  port: registry
              readinessProbe:
                httpGet:
                  path: /
                  port: registry
            volumes:
            - name: image-store
              persistentVolumeClaim:
                claimName: cephfs-pvc
                readOnly: false
  </code></pre>
  <p><strong>Storagelass.yaml</strong></p>
  <pre><code>        apiVersion: storage.k8s.io/v1
          kind: StorageClass
          metadata:
            name: rook-cephfs
          # Change &quot;rook-ceph&quot; provisioner prefix to match the operator namespace if needed
          provisioner: rook-ceph.cephfs.csi.ceph.com
          parameters:
            # clusterID is the namespace where operator is deployed.
            clusterID: rook-ceph
            # CephFS filesystem name into which the volume shall be created
            fsName: myfs
            # Ceph pool into which the volume shall be created
            # Required for provisionVolume: &quot;true&quot;
            pool: myfs-data0
            # Root path of an existing CephFS volume
            # Required for provisionVolume: &quot;false&quot;
            # rootPath: /absolute/path
            # The secrets contain Ceph admin credentials. These are generated automatically by the operator
            # in the same namespace as the cluster.
            csi.storage.k8s.io/provisioner-secret-name: rook-csi-cephfs-provisioner
            csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
            csi.storage.k8s.io/node-stage-secret-name: rook-csi-cephfs-node
            csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
          reclaimPolicy: Deletea
  </code></pre>
  <p><strong>kubectl describe pods  --namespace=kube-system kube-registry-58659ff99b-j2b4d</strong></p>
  <pre><code>        Name:           kube-registry-58659ff99b-j2b4d
          Namespace:      kube-system
          Priority:       0
          Node:           minikube/192.168.99.212
          Start Time:     Wed, 25 Nov 2020 13:19:35 -0500
          Labels:         k8s-app=kube-registry
                          kubernetes.io/cluster-service=true
                          pod-template-hash=58659ff99b
          Annotations:    &lt;none&gt;
          Status:         Pending
          IP:
          IPs:            &lt;none&gt;
          Controlled By:  ReplicaSet/kube-registry-58659ff99b
          Containers:
            registry:
              Container ID:
              Image:          registry:2
              Image ID:
              Port:           5000/TCP
              Host Port:      0/TCP
              State:          Waiting
                Reason:       ContainerCreating
              Ready:          False
              Restart Count:  0
              Limits:
                cpu:     100m
                memory:  100Mi
              Requests:
                cpu:      100m
                memory:   100Mi
              Liveness:   http-get http://:registry/ delay=0s timeout=1s period=10s #success=1 #failure=3
              Readiness:  http-get http://:registry/ delay=0s timeout=1s period=10s #success=1 #failure=3
              Environment:
                REGISTRY_HTTP_ADDR:                         :5000
                REGISTRY_HTTP_SECRET:                       Ple4seCh4ngeThisN0tAVerySecretV4lue
                REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY:  /var/lib/registry
              Mounts:
                /var/lib/registry from image-store (rw)
                /var/run/secrets/kubernetes.io/serviceaccount from default-token-nw4th (ro)
          Conditions:
            Type              Status
            Initialized       True
            Ready             False
            ContainersReady   False
            PodScheduled      True
          Volumes:
            image-store:
              Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
              ClaimName:  cephfs-pvc
              ReadOnly:   false
            default-token-nw4th:
              Type:        Secret (a volume populated by a Secret)
              SecretName:  default-token-nw4th
              Optional:    false
          QoS Class:       Guaranteed
          Node-Selectors:  &lt;none&gt;
          Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                           node.kubernetes.io/unreachable:NoExecute for 300s
          Events:
            Type     Reason              Age                 From                     Message
            ----     ------              ----                ----                     -------
            Warning  FailedScheduling    13m (x3 over 13m)   default-scheduler        running &quot;VolumeBinding&quot; filter plugin for pod &quot;kube-registry-58659ff99b-j2b4d&quot;: pod has unbound immediate PersistentVolumeClaims
            Normal   Scheduled           13m                 default-scheduler        Successfully assigned kube-system/kube-registry-58659ff99b-j2b4d to minikube
            Warning  FailedMount         2m6s (x5 over 11m)  kubelet, minikube        Unable to attach or mount volumes: unmounted volumes=[image-store], unattached volumes=[image-store default-token-nw4th]: timed out waiting for the condition
            Warning  FailedAttachVolume  59s (x6 over 11m)   attachdetach-controller  AttachVolume.Attach failed for volume &quot;pvc-6eeff481-eb0a-4269-84c7-e744c9d639d9&quot; : attachdetachment timeout for volume 0001-0009-rook-c
  </code></pre>
  <p><strong>ceph provisioner logs, I restarted my cluster so the name will be different but output is the same</strong></p>
  <pre><code>        I1127 18:27:19.370543       1 csi-provisioner.go:121] Version: v2.0.0
          I1127 18:27:19.370948       1 csi-provisioner.go:135] Building kube configs for running in cluster...
          I1127 18:27:19.429190       1 connection.go:153] Connecting to unix:///csi/csi-provisioner.sock
          I1127 18:27:21.561133       1 common.go:111] Probing CSI driver for readiness
          W1127 18:27:21.905396       1 metrics.go:142] metrics endpoint will not be started because `metrics-address` was not specified.
          I1127 18:27:22.060963       1 leaderelection.go:243] attempting to acquire leader lease  rook-ceph/rook-ceph-cephfs-csi-ceph-com...
          I1127 18:27:22.122303       1 leaderelection.go:253] successfully acquired lease rook-ceph/rook-ceph-cephfs-csi-ceph-com
          I1127 18:27:22.323990       1 controller.go:820] Starting provisioner controller rook-ceph.cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-797b67c54b-42jwc_4e14295b-f73d-4b94-bae9-ff4f2639b487!
          I1127 18:27:22.324061       1 clone_controller.go:66] Starting CloningProtection controller
          I1127 18:27:22.324205       1 clone_controller.go:84] Started CloningProtection controller
          I1127 18:27:22.325240       1 volume_store.go:97] Starting save volume queue
          I1127 18:27:22.426790       1 controller.go:869] Started provisioner controller rook-ceph.cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-797b67c54b-42jwc_4e14295b-f73d-4b94-bae9-ff4f2639b487!
          I1127 19:08:39.850493       1 controller.go:1317] provision &quot;kube-system/cephfs-pvc&quot; class &quot;rook-cephfs&quot;: started
          I1127 19:08:39.851034       1 event.go:282] Event(v1.ObjectReference{Kind:&quot;PersistentVolumeClaim&quot;, Namespace:&quot;kube-system&quot;, Name:&quot;cephfs-pvc&quot;, UID:&quot;7c47bda7-0c7b-4ca0-b6d0-19d717ef2e06&quot;, APIVersion:&quot;v1&quot;, ResourceVersion:&quot;7744&quot;, FieldPath:&quot;&quot;}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim &quot;kube-system/cephfs-pvc&quot;
          I1127 19:08:43.670226       1 controller.go:1420] provision &quot;kube-system/cephfs-pvc&quot; class &quot;rook-cephfs&quot;: volume &quot;pvc-7c47bda7-0c7b-4ca0-b6d0-19d717ef2e06&quot; provisioned
          I1127 19:08:43.670262       1 controller.go:1437] provision &quot;kube-system/cephfs-pvc&quot; class &quot;rook-cephfs&quot;: succeeded
          E1127 19:08:43.692108       1 controller.go:1443] couldn't create key for object pvc-7c47bda7-0c7b-4ca0-b6d0-19d717ef2e06: object has no meta: object does not implement the Object interfaces
          I1127 19:08:43.692189       1 controller.go:1317] provision &quot;kube-system/cephfs-pvc&quot; class &quot;rook-cephfs&quot;: started
          I1127 19:08:43.692205       1 controller.go:1326] provision &quot;kube-system/cephfs-pvc&quot; class &quot;rook-cephfs&quot;: persistentvolume &quot;pvc-7c47bda7-0c7b-4ca0-b6d0-19d717ef2e06&quot; already exists, skipping
          I1127 19:08:43.692220       1 event.go:282] Event(v1.ObjectReference{Kind:&quot;PersistentVolumeClaim&quot;, Namespace:&quot;kube-system&quot;, Name:&quot;cephfs-pvc&quot;, UID:&quot;7c47bda7-0c7b-4ca0-b6d0-19d717ef2e06&quot;, APIVersion:&quot;v1&quot;, ResourceVersion:&quot;7744&quot;, FieldPath:&quot;&quot;}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned 
  </code></pre>