title: Kubernetes Cinder volumes do not mount with cloud-provider=openstack
body: <p>I am trying to use the cinder plugin for kubernetes to create both statically defined PVs as well as StorageClasses, but I see no activity between my cluster and cinder for creating/mounting the devices.</p>
  
  <p>Kubernetes Version:</p>
  
  <pre><code>kubectl version
  Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.1", GitCommit:"33cf7b9acbb2cb7c9c72a10d6636321fb180b159", GitTreeState:"clean", BuildDate:"2016-10-10T18:19:49Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.1", GitCommit:"33cf7b9acbb2cb7c9c72a10d6636321fb180b159", GitTreeState:"clean", BuildDate:"2016-10-10T18:13:36Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}
  </code></pre>
  
  <p>The command kubelet was started with and its status:</p>
  
  <pre><code>systemctl status kubelet -l
  ● kubelet.service - Kubelet service
     Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
     Active: active (running) since Thu 2016-10-20 07:43:07 PDT; 3h 53min ago
    Process: 2406 ExecStartPre=/usr/local/bin/install-kube-binaries (code=exited, status=0/SUCCESS)
    Process: 2400 ExecStartPre=/usr/local/bin/create-certs (code=exited, status=0/SUCCESS)
   Main PID: 2408 (kubelet)
     CGroup: /system.slice/kubelet.service
             ├─2408 /usr/local/bin/kubelet --pod-manifest-path=/etc/kubernetes/manifests --api-servers=https://172.17.0.101:6443 --logtostderr=true --v=12 --allow-privileged=true --hostname-override=jk-kube2-master --pod-infra-container-image=pause-amd64:3.0 --cluster-dns=172.31.53.53 --cluster-domain=occloud --cloud-provider=openstack --cloud-config=/etc/cloud.conf
  </code></pre>
  
  <p>Here is my cloud.conf file:</p>
  
  <pre><code># cat /etc/cloud.conf
  [Global]
  username=&lt;user&gt;
  password=XXXXXXXX
  auth-url=http://&lt;openStack URL&gt;:5000/v2.0
  tenant-name=Shadow
  region=RegionOne
  </code></pre>
  
  <p>It appears that k8s is able to communicate successfully with openstack.  From /var/log/messages:</p>
  
  <pre><code>kubelet: I1020 11:43:51.770948    2408 openstack_instances.go:41] openstack.Instances() called
  kubelet: I1020 11:43:51.836642    2408 openstack_instances.go:78] Found 39 compute flavors
  kubelet: I1020 11:43:51.836679    2408 openstack_instances.go:79] Claiming to support Instances
  kubelet: I1020 11:43:51.836688    2408 openstack_instances.go:124] NodeAddresses(jk-kube2-master) called
  kubelet: I1020 11:43:52.274332    2408 openstack_instances.go:131] NodeAddresses(jk-kube2-master) =&gt; [{InternalIP 172.17.0.101} {ExternalIP 10.75.152.101}]
  </code></pre>
  
  <p>My PV/PVC yaml files, and cinder list output:</p>
  
  <pre><code># cat persistentVolume.yaml
  kind: PersistentVolume
  apiVersion: v1
  metadata:
    name: jk-test
    labels:
      type: test
  spec:
    capacity:
      storage: 10Gi
    accessModes:
      - ReadWriteOnce
    cinder:
      volumeID: 48d2d1e6-e063-437a-855f-8b62b640a950
      fsType: ext4
  
  # cat persistentVolumeClaim.yaml
  kind: PersistentVolumeClaim
  apiVersion: v1
  metadata:
    name: myclaim
  spec:
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 10Gi
    selector:
      matchLabels:
        type: "test"
  # cinder list | grep jk-cinder
  | 48d2d1e6-e063-437a-855f-8b62b640a950 | available |              jk-cinder              |  10  |      -      |  false   |          
  </code></pre>
  
  <p>As seen above, cinder reports the device with the ID referenced in the pv.yaml file is available.  When I create them, things seem to work:</p>
  
  <pre><code>NAME         CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS    CLAIM             REASON    AGE
  pv/jk-test   10Gi       RWO           Retain          Bound     default/myclaim             5h
  NAME               STATUS    VOLUME    CAPACITY   ACCESSMODES   AGE
  pvc/myclaim        Bound     jk-test   10Gi       RWO           5h
  </code></pre>
  
  <p>Then I try to create a pod using the pvc, but it fails to mount the volume:</p>
  
  <pre><code># cat testPod.yaml
  kind: Pod
  apiVersion: v1
  metadata:
    name: jk-test3
    labels:
      name: jk-test
  spec:
    containers:
      - name: front-end
        image: example-front-end:latest
        ports:
          - hostPort: 6000
            containerPort: 3000
    volumes:
      - name: jk-test
        persistentVolumeClaim:
          claimName: myclaim
  </code></pre>
  
  <p>And here is the state of the pod:</p>
  
  <pre><code>  3h            46s             109     {kubelet jk-kube2-master}                       Warning         FailedMount     Unable to mount volumes for pod "jk-test3_default(0f83368f-96d4-11e6-8243-fa163ebfcd23)": timeout expired waiting for volumes to attach/mount for pod "jk-test3"/"default". list of unattached/unmounted volumes=[jk-test]
    3h            46s             109     {kubelet jk-kube2-master}                       Warning         FailedSync      Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "jk-test3"/"default". list of unattached/unmounted volumes=[jk-test]
  </code></pre>
  
  <p>I've verified that my openstack provider is exposing cinder v1 and v2 APIs and the previous logs from openstack_instances show the nova API is accessible.  Despite that, I never see any attempts on k8s part to communicate with cinder or nova to mount the volume.</p>
  
  <p>Here are what I think are the relevant log messages regarding the failure to mount:</p>
  
  <pre><code>kubelet: I1020 06:51:11.840341   24027 desired_state_of_world_populator.go:323] Extracted volumeSpec (0x23a45e0) from bound PV (pvName "jk-test") and PVC (ClaimName "default"/"myclaim" pvcUID 51919dfb-96c9-11e6-8243-fa163ebfcd23)
  kubelet: I1020 06:51:11.840424   24027 desired_state_of_world_populator.go:241] Added volume "jk-test" (volSpec="jk-test") for pod "f957f140-96cb-11e6-8243-fa163ebfcd23" to desired state.
  kubelet: I1020 06:51:11.840474   24027 desired_state_of_world_populator.go:241] Added volume "default-token-js40f" (volSpec="default-token-js40f") for pod "f957f140-96cb-11e6-8243-fa163ebfcd23" to desired state.
  kubelet: I1020 06:51:11.896176   24027 reconciler.go:201] Attempting to start VerifyControllerAttachedVolume for volume "kubernetes.io/cinder/48d2d1e6-e063-437a-855f-8b62b640a950" (spec.Name: "jk-test") pod "f957f140-96cb-11e6-8243-fa163ebfcd23" (UID: "f957f140-96cb-11e6-8243-fa163ebfcd23")
  kubelet: I1020 06:51:11.896330   24027 reconciler.go:225] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/cinder/48d2d1e6-e063-437a-855f-8b62b640a950" (spec.Name: "jk-test") pod "f957f140-96cb-11e6-8243-fa163ebfcd23" (UID: "f957f140-96cb-11e6-8243-fa163ebfcd23")
  kubelet: I1020 06:51:11.896361   24027 reconciler.go:201] Attempting to start VerifyControllerAttachedVolume for volume "kubernetes.io/secret/f957f140-96cb-11e6-8243-fa163ebfcd23-default-token-js40f" (spec.Name: "default-token-js40f") pod "f957f140-96cb-11e6-8243-fa163ebfcd23" (UID: "f957f140-96cb-11e6-8243-fa163ebfcd23")
  kubelet: I1020 06:51:11.896390   24027 reconciler.go:225] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/secret/f957f140-96cb-11e6-8243-fa163ebfcd23-default-token-js40f" (spec.Name: "default-token-js40f") pod "f957f140-96cb-11e6-8243-fa163ebfcd23" (UID: "f957f140-96cb-11e6-8243-fa163ebfcd23")
  kubelet: I1020 06:51:11.896420   24027 config.go:98] Looking for [api file], have seen map[file:{} api:{}]
  kubelet: E1020 06:51:11.896566   24027 nestedpendingoperations.go:253] Operation for "\"kubernetes.io/cinder/48d2d1e6-e063-437a-855f-8b62b640a950\"" failed. No retries permitted until 2016-10-20 06:53:11.896529189 -0700 PDT (durationBeforeRetry 2m0s). Error: Volume "kubernetes.io/cinder/48d2d1e6-e063-437a-855f-8b62b640a950" (spec.Name: "jk-test") pod "f957f140-96cb-11e6-8243-fa163ebfcd23" (UID: "f957f140-96cb-11e6-8243-fa163ebfcd23") has not yet been added to the list of VolumesInUse in the node's volume status.
  </code></pre>
  
  <p>Is there a piece I am missing?  I've followed the instructions here: <a href="https://github.com/kubernetes/kubernetes/tree/master/examples/mysql-cinder-pd" rel="nofollow">k8s - mysql-cinder-pd example</a> But haven't been able to get any communication.  As another datapoint I tried defining a Storage class as provided by k8s, here are the associated StorageClass and PVC files:</p>
  
  <pre><code># cat cinderStorage.yaml
  kind: StorageClass
  apiVersion: storage.k8s.io/v1beta1
  metadata:
    name: gold
  provisioner: kubernetes.io/cinder
  parameters:
    availability: nova
  # cat dynamicPVC.yaml
  kind: PersistentVolumeClaim
  apiVersion: v1
  metadata:
    name: dynamicclaim
    annotations:
      volume.beta.kubernetes.io/storage-class: "gold"
  spec:
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 8Gi
  </code></pre>
  
  <p>The StorageClass reports success, but when I try to create the PVC it gets stuck in the 'pending' state and reports 'no volume plugin matched':</p>
  
  <pre><code># kubectl get storageclass
  NAME      TYPE
  gold      kubernetes.io/cinder
  # kubectl describe pvc dynamicclaim
  Name:           dynamicclaim
  Namespace:      default
  Status:         Pending
  Volume:
  Labels:         &lt;none&gt;
  Capacity:
  Access Modes:
  Events:
    FirstSeen     LastSeen        Count   From                            SubobjectPath   Type            Reason                  Message
    ---------     --------        -----   ----                            -------------   --------        ------                  -------
    1d            15s             5867    {persistentvolume-controller }                  Warning         ProvisioningFailed      no volume plugin matched
  </code></pre>
  
  <p>This contradicts whats in the logs for plugins that were loaded:</p>
  
  <pre><code>grep plugins /var/log/messages
  kubelet: I1019 11:39:41.382517   22435 plugins.go:56] Registering credential provider: .dockercfg
  kubelet: I1019 11:39:41.382673   22435 plugins.go:355] Loaded volume plugin "kubernetes.io/aws-ebs"
  kubelet: I1019 11:39:41.382685   22435 plugins.go:355] Loaded volume plugin "kubernetes.io/empty-dir"
  kubelet: I1019 11:39:41.382691   22435 plugins.go:355] Loaded volume plugin "kubernetes.io/gce-pd"
  kubelet: I1019 11:39:41.382698   22435 plugins.go:355] Loaded volume plugin "kubernetes.io/git-repo"
  kubelet: I1019 11:39:41.382705   22435 plugins.go:355] Loaded volume plugin "kubernetes.io/host-path"
  kubelet: I1019 11:39:41.382712   22435 plugins.go:355] Loaded volume plugin "kubernetes.io/nfs"
  kubelet: I1019 11:39:41.382718   22435 plugins.go:355] Loaded volume plugin "kubernetes.io/secret"
  kubelet: I1019 11:39:41.382725   22435 plugins.go:355] Loaded volume plugin "kubernetes.io/iscsi"
  kubelet: I1019 11:39:41.382734   22435 plugins.go:355] Loaded volume plugin "kubernetes.io/glusterfs"
  jk-kube2-master kubelet: I1019 11:39:41.382741   22435 plugins.go:355] Loaded volume plugin "kubernetes.io/rbd"
  kubelet: I1019 11:39:41.382749   22435 plugins.go:355] Loaded volume plugin "kubernetes.io/cinder"
  kubelet: I1019 11:39:41.382755   22435 plugins.go:355] Loaded volume plugin "kubernetes.io/quobyte"
  kubelet: I1019 11:39:41.382762   22435 plugins.go:355] Loaded volume plugin "kubernetes.io/cephfs"
  kubelet: I1019 11:39:41.382781   22435 plugins.go:355] Loaded volume plugin "kubernetes.io/downward-api"
  kubelet: I1019 11:39:41.382798   22435 plugins.go:355] Loaded volume plugin "kubernetes.io/fc"
  kubelet: I1019 11:39:41.382804   22435 plugins.go:355] Loaded volume plugin "kubernetes.io/flocker"
  kubelet: I1019 11:39:41.382822   22435 plugins.go:355] Loaded volume plugin "kubernetes.io/azure-file"
  kubelet: I1019 11:39:41.382839   22435 plugins.go:355] Loaded volume plugin "kubernetes.io/configmap"
  kubelet: I1019 11:39:41.382846   22435 plugins.go:355] Loaded volume plugin "kubernetes.io/vsphere-volume"
  kubelet: I1019 11:39:41.382853   22435 plugins.go:355] Loaded volume plugin "kubernetes.io/azure-disk"
  </code></pre>
  
  <p>And I have the nova and cinder clients installed on my machine:</p>
  
  <pre><code># which nova
  /usr/bin/nova
  # which cinder
  /usr/bin/cinder
  </code></pre>
  
  <p>Any help is appreciated, I'm sure I'm missing something simple here.</p>
  
  <p>Thanks!</p>