<p>I can't mount GCE PersistentVolumes using Kubernetes 1.8.0, each POD are stuck in ContainerCreating state.</p>

<p>This output is from a test environment I put up for this lab. Worth to mention is that I'm using Compute Engine, NOT Kubernetes Engine.</p>

<p>I have not configured any cloud settings and I wounder if this might be the root cause but gcloud works perfectly fine from the worker and all my VMs in this lab environment are allowed full access to the API. </p>

<p>Error message on the worker</p>

<pre><code>Jan  2 13:03:58 worker-0 kubelet[1421]: E0102 13:03:58.733299    1421 kubelet.go:1628] Unable to mount volumes for pod "mysql-cgui-01-5c85f7dd86-gt2s8_default(ab17eaf2-efb6-11e7-a385-42010af0000a)": timeout expired waiting for volumes to attach/mount for pod "default"/"mysql-cgui-01-5c85f7dd86-gt2s8". list of unattached/unmounted volumes=[mysql-cgui-01]; skipping pod
</code></pre>

<p>POD description</p>

<pre><code>bofh:~$ kubectl describe pod mysql-cgui-01-5c85f7dd86-gt2s8
Name:           mysql-cgui-01-5c85f7dd86-gt2s8
Namespace:      default
Node:           worker-0/10.240.0.20
Start Time:     Tue, 02 Jan 2018 12:15:49 +0000
Labels:         name=mysql-cgui-01
                pod-template-hash=1741938842
Annotations:    kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"default","name":"mysql-cgui-01-5c85f7dd86","uid":"ab10f9ef-efb6-11e7-a385-42010af...
Status:         Pending
IP:
Created By:     ReplicaSet/mysql-cgui-01-5c85f7dd86
Controlled By:  ReplicaSet/mysql-cgui-01-5c85f7dd86
Containers:
  mysql-cgui-01:
    Container ID:
    Image:          external/mysql:latest
    Image ID:
    Port:           3306/TCP
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Mounts:
      /data/mysql from mysql-cgui-01 (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-tb6sm (ro)
Conditions:
  Type           Status
  Initialized    True
  Ready          False
  PodScheduled   True
Volumes:
  mysql-cgui-01:
    Type:       GCEPersistentDisk (a Persistent Disk resource in Google Compute Engine)
    PDName:     mysql-cgui-01
    FSType:     ext4
    Partition:  0
    ReadOnly:   false
  default-token-tb6sm:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-tb6sm
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  &lt;none&gt;
Tolerations:     &lt;none&gt;
Events:
  Type     Reason                 Age                From               Message
  ----     ------                 ----               ----               -------
  Normal   Scheduled              55m                default-scheduler  Successfully assigned mysql-cgui-01-5c85f7dd86-gt2s8 to worker-0
  Normal   SuccessfulMountVolume  55m                kubelet, worker-0  MountVolume.SetUp succeeded for volume "default-token-tb6sm"
  Warning  FailedMount            41m (x6 over 53m)  kubelet, worker-0  Unable to mount volumes for pod "mysql-cgui-01-5c85f7dd86-gt2s8_default(ab17eaf2-efb6-11e7-a385-42010af0000a)": timeout expired waiting for volumes to attach/mount for pod "default"/"mysql-cgui-01-5c85f7dd86-gt2s8". list of unattached/unmounted volumes=[mysql-cgui-01]
  Warning  FailedSync             41m (x6 over 53m)  kubelet, worker-0  Error syncing pod
  Normal   SuccessfulMountVolume  38m                kubelet, worker-0  MountVolume.SetUp succeeded for volume "default-token-tb6sm"
  Warning  FailedMount            4m (x15 over 36m)  kubelet, worker-0  Unable to mount volumes for pod "mysql-cgui-01-5c85f7dd86-gt2s8_default(ab17eaf2-efb6-11e7-a385-42010af0000a)": timeout expired waiting for volumes to attach/mount for pod "default"/"mysql-cgui-01-5c85f7dd86-gt2s8". list of unattached/unmounted volumes=[mysql-cgui-01]
  Warning  FailedSync             2m (x16 over 36m)  kubelet, worker-0  Error syncing pod
</code></pre>

<p>Testing using gcloud from worker-0</p>

<pre><code>worker-0:~$ gcloud compute disks list
NAME           ZONE            SIZE_GB  TYPE         STATUS
bofh           europe-west1-d  20       pd-standard  READY
controller-0   europe-west1-c  200      pd-standard  READY
controller-1   europe-west1-c  200      pd-standard  READY
controller-2   europe-west1-c  200      pd-standard  READY
mysql-cgui-01  europe-west1-c  10       pd-standard  READY
mysql-cgui-02  europe-west1-c  10       pd-standard  READY
worker-0       europe-west1-c  200      pd-standard  READY
worker-1       europe-west1-c  200      pd-standard  READY
worker-2       europe-west1-c  200      pd-standard  READY
</code></pre>

<p>Worker-0 kubelet flags</p>

<pre><code>ExecStart=/usr/local/bin/kubelet \
  --allow-privileged=true \
  --anonymous-auth=false \
  --authorization-mode=Webhook \
  --client-ca-file=/var/lib/kubernetes/ca.pem \
  --cluster-dns=10.32.0.10 \
  --cluster-domain=cluster.local \
  --container-runtime=docker \
  --image-pull-progress-deadline=2m \
  --kubeconfig=/var/lib/kubelet/kubeconfig \
  --network-plugin=cni \
  --pod-cidr=10.200.0.0/24 \
  --register-node=true \
  --require-kubeconfig \
  --runtime-request-timeout=15m \
  --tls-cert-file=/var/lib/kubelet/worker-0.pem \
  --tls-private-key-file=/var/lib/kubelet/worker-0-key.pem \
  --cloud-provider=gce \
  --v=2
</code></pre>

<p>Worker-0 kube-proxy flags</p>

<pre><code>ExecStart=/usr/local/bin/kube-proxy \
  --cluster-cidr=10.200.0.0/16 \
  --kubeconfig=/var/lib/kube-proxy/kubeconfig \
  --proxy-mode=iptables \
  --v=2
</code></pre>

<p>Controller kube-scheduler flags</p>

<pre><code>ExecStart=/usr/local/bin/kube-scheduler \
  --leader-elect=true \
  --master=http://127.0.0.1:8080 \
  --v=2
</code></pre>

<p>Controller kube-controllermanager flags</p>

<pre><code>ExecStart=/usr/local/bin/kube-controller-manager \
  --address=0.0.0.0 \
  --cluster-cidr=10.200.0.0/16 \
  --cluster-name=kubernetes \
  --cluster-signing-cert-file=/var/lib/kubernetes/ca.pem \
  --cluster-signing-key-file=/var/lib/kubernetes/ca-key.pem \
  --leader-elect=true \
  --master=http://127.0.0.1:8080 \
  --root-ca-file=/var/lib/kubernetes/ca.pem \
  --service-account-private-key-file=/var/lib/kubernetes/ca-key.pem \
  --service-cluster-ip-range=10.32.0.0/24 \
  --cloud-provider=gce \
  --allocate-node-cidrs=false \
  --configure-cloud-routes=false \
  --v=2
</code></pre>

<p>controller kube-apiserver flags</p>

<pre><code>ExecStart=/usr/local/bin/kube-apiserver \
  --admission-control=Initializers,NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \
  --advertise-address=10.240.0.10 \
  --allow-privileged=true \
  --apiserver-count=3 \
  --audit-log-maxage=30 \
  --audit-log-maxbackup=3 \
  --audit-log-maxsize=100 \
  --audit-log-path=/var/log/audit.log \
  --authorization-mode=Node,RBAC \
  --bind-address=0.0.0.0 \
  --client-ca-file=/var/lib/kubernetes/ca.pem \
  --enable-swagger-ui=true \
  --etcd-cafile=/var/lib/kubernetes/ca.pem \
  --etcd-certfile=/var/lib/kubernetes/kubernetes.pem \
  --etcd-keyfile=/var/lib/kubernetes/kubernetes-key.pem \
  --etcd-servers=https://10.240.0.10:2379,https://10.240.0.11:2379,https://10.240.0.12:2379 \
  --event-ttl=1h \
  --experimental-encryption-provider-config=/var/lib/kubernetes/encryption-config.yaml \
  --insecure-bind-address=127.0.0.1 \
  --kubelet-certificate-authority=/var/lib/kubernetes/ca.pem \
  --kubelet-client-certificate=/var/lib/kubernetes/kubernetes.pem \
  --kubelet-client-key=/var/lib/kubernetes/kubernetes-key.pem \
  --kubelet-https=true \
  --runtime-config=api/all \
  --service-account-key-file=/var/lib/kubernetes/ca-key.pem \
  --service-cluster-ip-range=10.32.0.0/24 \
  --service-node-port-range=30000-32767 \
  --tls-ca-file=/var/lib/kubernetes/ca.pem \
  --tls-cert-file=/var/lib/kubernetes/kubernetes.pem \
  --tls-private-key-file=/var/lib/kubernetes/kubernetes-key.pem \
  --cloud-provider=gce \
  --v=2
</code></pre>

<p>Versions</p>

<pre><code>bofh:~$ kubectl version
Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"6e937839ac04a38cac63e6a7a306c5d035fe7b0a", GitTreeState:"clean", BuildDate:"2017-09-28T22:57:57Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"6e937839ac04a38cac63e6a7a306c5d035fe7b0a", GitTreeState:"clean", BuildDate:"2017-09-28T22:46:41Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
bofh:~$ kubectl get nodes
NAME       STATUS    ROLES     AGE       VERSION
worker-0   Ready     &lt;none&gt;    12d       v1.8.0
worker-1   Ready     &lt;none&gt;    12d       v1.8.0
worker-2   Ready     &lt;none&gt;    12d       v1.8.0
</code></pre>

<p>PV and PVC info</p>

<pre><code>bofh:~$ kubectl get pv
No resources found.
bofh:~$ kubectl get pvc
No resources found.
bofh:~$ kubectl describe pv
bofh:~$ kubectl describe pvc
</code></pre>

<p>StorageClass config</p>

<pre><code>kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: standard
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-standard
</code></pre>

<p>Storage class output</p>

<pre><code>bofh:~$ kubectl get sc
NAME       PROVISIONER
slow       kubernetes.io/gce-pd
standard   kubernetes.io/gce-pd
</code></pre>

<p>You need to add the cloud-provider flag to the apiserver, kubelet and controller-manager.</p>

<p>You also then need to configure a storageclass for the GCE</p>
