title: Kubernetes POD Timeout expired waiting for volumes to attach/mount
body: <p>I can't mount GCE PersistentVolumes using Kubernetes 1.8.0, each POD are stuck in ContainerCreating state.</p>
  
  <p>This output is from a test environment I put up for this lab. Worth to mention is that I'm using Compute Engine, NOT Kubernetes Engine.</p>
  
  <p>I have not configured any cloud settings and I wounder if this might be the root cause but gcloud works perfectly fine from the worker and all my VMs in this lab environment are allowed full access to the API. </p>
  
  <p>Error message on the worker</p>
  
  <pre><code>Jan  2 13:03:58 worker-0 kubelet[1421]: E0102 13:03:58.733299    1421 kubelet.go:1628] Unable to mount volumes for pod "mysql-cgui-01-5c85f7dd86-gt2s8_default(ab17eaf2-efb6-11e7-a385-42010af0000a)": timeout expired waiting for volumes to attach/mount for pod "default"/"mysql-cgui-01-5c85f7dd86-gt2s8". list of unattached/unmounted volumes=[mysql-cgui-01]; skipping pod
  </code></pre>
  
  <p>POD description</p>
  
  <pre><code>bofh:~$ kubectl describe pod mysql-cgui-01-5c85f7dd86-gt2s8
  Name:           mysql-cgui-01-5c85f7dd86-gt2s8
  Namespace:      default
  Node:           worker-0/10.240.0.20
  Start Time:     Tue, 02 Jan 2018 12:15:49 +0000
  Labels:         name=mysql-cgui-01
                  pod-template-hash=1741938842
  Annotations:    kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"default","name":"mysql-cgui-01-5c85f7dd86","uid":"ab10f9ef-efb6-11e7-a385-42010af...
  Status:         Pending
  IP:
  Created By:     ReplicaSet/mysql-cgui-01-5c85f7dd86
  Controlled By:  ReplicaSet/mysql-cgui-01-5c85f7dd86
  Containers:
    mysql-cgui-01:
      Container ID:
      Image:          external/mysql:latest
      Image ID:
      Port:           3306/TCP
      State:          Waiting
        Reason:       ContainerCreating
      Ready:          False
      Restart Count:  0
      Mounts:
        /data/mysql from mysql-cgui-01 (rw)
        /var/run/secrets/kubernetes.io/serviceaccount from default-token-tb6sm (ro)
  Conditions:
    Type           Status
    Initialized    True
    Ready          False
    PodScheduled   True
  Volumes:
    mysql-cgui-01:
      Type:       GCEPersistentDisk (a Persistent Disk resource in Google Compute Engine)
      PDName:     mysql-cgui-01
      FSType:     ext4
      Partition:  0
      ReadOnly:   false
    default-token-tb6sm:
      Type:        Secret (a volume populated by a Secret)
      SecretName:  default-token-tb6sm
      Optional:    false
  QoS Class:       BestEffort
  Node-Selectors:  &lt;none&gt;
  Tolerations:     &lt;none&gt;
  Events:
    Type     Reason                 Age                From               Message
    ----     ------                 ----               ----               -------
    Normal   Scheduled              55m                default-scheduler  Successfully assigned mysql-cgui-01-5c85f7dd86-gt2s8 to worker-0
    Normal   SuccessfulMountVolume  55m                kubelet, worker-0  MountVolume.SetUp succeeded for volume "default-token-tb6sm"
    Warning  FailedMount            41m (x6 over 53m)  kubelet, worker-0  Unable to mount volumes for pod "mysql-cgui-01-5c85f7dd86-gt2s8_default(ab17eaf2-efb6-11e7-a385-42010af0000a)": timeout expired waiting for volumes to attach/mount for pod "default"/"mysql-cgui-01-5c85f7dd86-gt2s8". list of unattached/unmounted volumes=[mysql-cgui-01]
    Warning  FailedSync             41m (x6 over 53m)  kubelet, worker-0  Error syncing pod
    Normal   SuccessfulMountVolume  38m                kubelet, worker-0  MountVolume.SetUp succeeded for volume "default-token-tb6sm"
    Warning  FailedMount            4m (x15 over 36m)  kubelet, worker-0  Unable to mount volumes for pod "mysql-cgui-01-5c85f7dd86-gt2s8_default(ab17eaf2-efb6-11e7-a385-42010af0000a)": timeout expired waiting for volumes to attach/mount for pod "default"/"mysql-cgui-01-5c85f7dd86-gt2s8". list of unattached/unmounted volumes=[mysql-cgui-01]
    Warning  FailedSync             2m (x16 over 36m)  kubelet, worker-0  Error syncing pod
  </code></pre>
  
  <p>Testing using gcloud from worker-0</p>
  
  <pre><code>worker-0:~$ gcloud compute disks list
  NAME           ZONE            SIZE_GB  TYPE         STATUS
  bofh           europe-west1-d  20       pd-standard  READY
  controller-0   europe-west1-c  200      pd-standard  READY
  controller-1   europe-west1-c  200      pd-standard  READY
  controller-2   europe-west1-c  200      pd-standard  READY
  mysql-cgui-01  europe-west1-c  10       pd-standard  READY
  mysql-cgui-02  europe-west1-c  10       pd-standard  READY
  worker-0       europe-west1-c  200      pd-standard  READY
  worker-1       europe-west1-c  200      pd-standard  READY
  worker-2       europe-west1-c  200      pd-standard  READY
  </code></pre>
  
  <p>Worker-0 kubelet flags</p>
  
  <pre><code>ExecStart=/usr/local/bin/kubelet \
    --allow-privileged=true \
    --anonymous-auth=false \
    --authorization-mode=Webhook \
    --client-ca-file=/var/lib/kubernetes/ca.pem \
    --cluster-dns=10.32.0.10 \
    --cluster-domain=cluster.local \
    --container-runtime=docker \
    --image-pull-progress-deadline=2m \
    --kubeconfig=/var/lib/kubelet/kubeconfig \
    --network-plugin=cni \
    --pod-cidr=10.200.0.0/24 \
    --register-node=true \
    --require-kubeconfig \
    --runtime-request-timeout=15m \
    --tls-cert-file=/var/lib/kubelet/worker-0.pem \
    --tls-private-key-file=/var/lib/kubelet/worker-0-key.pem \
    --cloud-provider=gce \
    --v=2
  </code></pre>
  
  <p>Worker-0 kube-proxy flags</p>
  
  <pre><code>ExecStart=/usr/local/bin/kube-proxy \
    --cluster-cidr=10.200.0.0/16 \
    --kubeconfig=/var/lib/kube-proxy/kubeconfig \
    --proxy-mode=iptables \
    --v=2
  </code></pre>
  
  <p>Controller kube-scheduler flags</p>
  
  <pre><code>ExecStart=/usr/local/bin/kube-scheduler \
    --leader-elect=true \
    --master=http://127.0.0.1:8080 \
    --v=2
  </code></pre>
  
  <p>Controller kube-controllermanager flags</p>
  
  <pre><code>ExecStart=/usr/local/bin/kube-controller-manager \
    --address=0.0.0.0 \
    --cluster-cidr=10.200.0.0/16 \
    --cluster-name=kubernetes \
    --cluster-signing-cert-file=/var/lib/kubernetes/ca.pem \
    --cluster-signing-key-file=/var/lib/kubernetes/ca-key.pem \
    --leader-elect=true \
    --master=http://127.0.0.1:8080 \
    --root-ca-file=/var/lib/kubernetes/ca.pem \
    --service-account-private-key-file=/var/lib/kubernetes/ca-key.pem \
    --service-cluster-ip-range=10.32.0.0/24 \
    --cloud-provider=gce \
    --allocate-node-cidrs=false \
    --configure-cloud-routes=false \
    --v=2
  </code></pre>
  
  <p>controller kube-apiserver flags</p>
  
  <pre><code>ExecStart=/usr/local/bin/kube-apiserver \
    --admission-control=Initializers,NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \
    --advertise-address=10.240.0.10 \
    --allow-privileged=true \
    --apiserver-count=3 \
    --audit-log-maxage=30 \
    --audit-log-maxbackup=3 \
    --audit-log-maxsize=100 \
    --audit-log-path=/var/log/audit.log \
    --authorization-mode=Node,RBAC \
    --bind-address=0.0.0.0 \
    --client-ca-file=/var/lib/kubernetes/ca.pem \
    --enable-swagger-ui=true \
    --etcd-cafile=/var/lib/kubernetes/ca.pem \
    --etcd-certfile=/var/lib/kubernetes/kubernetes.pem \
    --etcd-keyfile=/var/lib/kubernetes/kubernetes-key.pem \
    --etcd-servers=https://10.240.0.10:2379,https://10.240.0.11:2379,https://10.240.0.12:2379 \
    --event-ttl=1h \
    --experimental-encryption-provider-config=/var/lib/kubernetes/encryption-config.yaml \
    --insecure-bind-address=127.0.0.1 \
    --kubelet-certificate-authority=/var/lib/kubernetes/ca.pem \
    --kubelet-client-certificate=/var/lib/kubernetes/kubernetes.pem \
    --kubelet-client-key=/var/lib/kubernetes/kubernetes-key.pem \
    --kubelet-https=true \
    --runtime-config=api/all \
    --service-account-key-file=/var/lib/kubernetes/ca-key.pem \
    --service-cluster-ip-range=10.32.0.0/24 \
    --service-node-port-range=30000-32767 \
    --tls-ca-file=/var/lib/kubernetes/ca.pem \
    --tls-cert-file=/var/lib/kubernetes/kubernetes.pem \
    --tls-private-key-file=/var/lib/kubernetes/kubernetes-key.pem \
    --cloud-provider=gce \
    --v=2
  </code></pre>
  
  <p>Versions</p>
  
  <pre><code>bofh:~$ kubectl version
  Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"6e937839ac04a38cac63e6a7a306c5d035fe7b0a", GitTreeState:"clean", BuildDate:"2017-09-28T22:57:57Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"6e937839ac04a38cac63e6a7a306c5d035fe7b0a", GitTreeState:"clean", BuildDate:"2017-09-28T22:46:41Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
  bofh:~$ kubectl get nodes
  NAME       STATUS    ROLES     AGE       VERSION
  worker-0   Ready     &lt;none&gt;    12d       v1.8.0
  worker-1   Ready     &lt;none&gt;    12d       v1.8.0
  worker-2   Ready     &lt;none&gt;    12d       v1.8.0
  </code></pre>
  
  <p>PV and PVC info</p>
  
  <pre><code>bofh:~$ kubectl get pv
  No resources found.
  bofh:~$ kubectl get pvc
  No resources found.
  bofh:~$ kubectl describe pv
  bofh:~$ kubectl describe pvc
  </code></pre>
  
  <p>StorageClass config</p>
  
  <pre><code>kind: StorageClass
  apiVersion: storage.k8s.io/v1
  metadata:
    name: standard
  provisioner: kubernetes.io/gce-pd
  parameters:
    type: pd-standard
  </code></pre>
  
  <p>Storage class output</p>
  
  <pre><code>bofh:~$ kubectl get sc
  NAME       PROVISIONER
  slow       kubernetes.io/gce-pd
  standard   kubernetes.io/gce-pd
  </code></pre>