{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/36067","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/36067/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/36067/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/36067/events","html_url":"https://github.com/kubernetes/kubernetes/issues/36067","id":186820703,"number":36067,"title":"Failing to mount GCE persistent disk","user":{"login":"autrilla","id":2260567,"avatar_url":"https://avatars.githubusercontent.com/u/2260567?v=3","gravatar_id":"","url":"https://api.github.com/users/autrilla","html_url":"https://github.com/autrilla","followers_url":"https://api.github.com/users/autrilla/followers","following_url":"https://api.github.com/users/autrilla/following{/other_user}","gists_url":"https://api.github.com/users/autrilla/gists{/gist_id}","starred_url":"https://api.github.com/users/autrilla/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/autrilla/subscriptions","organizations_url":"https://api.github.com/users/autrilla/orgs","repos_url":"https://api.github.com/users/autrilla/repos","events_url":"https://api.github.com/users/autrilla/events{/privacy}","received_events_url":"https://api.github.com/users/autrilla/received_events","type":"User","site_admin":false},"labels":[{"id":169428334,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/area/storage","name":"area/storage","color":"006b75","default":false},{"id":116719829,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/component/kubelet","name":"component/kubelet","color":"0052cc","default":false},{"id":114528068,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/priority/P0","name":"priority/P0","color":"e11d21","default":false},{"id":173493827,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/cluster","name":"team/cluster","color":"d2b48c","default":false}],"state":"open","locked":false,"assignee":{"login":"saad-ali","id":10052848,"avatar_url":"https://avatars.githubusercontent.com/u/10052848?v=3","gravatar_id":"","url":"https://api.github.com/users/saad-ali","html_url":"https://github.com/saad-ali","followers_url":"https://api.github.com/users/saad-ali/followers","following_url":"https://api.github.com/users/saad-ali/following{/other_user}","gists_url":"https://api.github.com/users/saad-ali/gists{/gist_id}","starred_url":"https://api.github.com/users/saad-ali/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/saad-ali/subscriptions","organizations_url":"https://api.github.com/users/saad-ali/orgs","repos_url":"https://api.github.com/users/saad-ali/repos","events_url":"https://api.github.com/users/saad-ali/events{/privacy}","received_events_url":"https://api.github.com/users/saad-ali/received_events","type":"User","site_admin":false},"assignees":[{"login":"saad-ali","id":10052848,"avatar_url":"https://avatars.githubusercontent.com/u/10052848?v=3","gravatar_id":"","url":"https://api.github.com/users/saad-ali","html_url":"https://github.com/saad-ali","followers_url":"https://api.github.com/users/saad-ali/followers","following_url":"https://api.github.com/users/saad-ali/following{/other_user}","gists_url":"https://api.github.com/users/saad-ali/gists{/gist_id}","starred_url":"https://api.github.com/users/saad-ali/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/saad-ali/subscriptions","organizations_url":"https://api.github.com/users/saad-ali/orgs","repos_url":"https://api.github.com/users/saad-ali/repos","events_url":"https://api.github.com/users/saad-ali/events{/privacy}","received_events_url":"https://api.github.com/users/saad-ali/received_events","type":"User","site_admin":false},{"login":"jingxu97","id":19172805,"avatar_url":"https://avatars.githubusercontent.com/u/19172805?v=3","gravatar_id":"","url":"https://api.github.com/users/jingxu97","html_url":"https://github.com/jingxu97","followers_url":"https://api.github.com/users/jingxu97/followers","following_url":"https://api.github.com/users/jingxu97/following{/other_user}","gists_url":"https://api.github.com/users/jingxu97/gists{/gist_id}","starred_url":"https://api.github.com/users/jingxu97/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jingxu97/subscriptions","organizations_url":"https://api.github.com/users/jingxu97/orgs","repos_url":"https://api.github.com/users/jingxu97/repos","events_url":"https://api.github.com/users/jingxu97/events{/privacy}","received_events_url":"https://api.github.com/users/jingxu97/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/23","html_url":"https://github.com/kubernetes/kubernetes/milestones/v1.5","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/23/labels","id":1945978,"number":23,"title":"v1.5","description":"","creator":{"login":"lavalamp","id":647318,"avatar_url":"https://avatars.githubusercontent.com/u/647318?v=3","gravatar_id":"","url":"https://api.github.com/users/lavalamp","html_url":"https://github.com/lavalamp","followers_url":"https://api.github.com/users/lavalamp/followers","following_url":"https://api.github.com/users/lavalamp/following{/other_user}","gists_url":"https://api.github.com/users/lavalamp/gists{/gist_id}","starred_url":"https://api.github.com/users/lavalamp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lavalamp/subscriptions","organizations_url":"https://api.github.com/users/lavalamp/orgs","repos_url":"https://api.github.com/users/lavalamp/repos","events_url":"https://api.github.com/users/lavalamp/events{/privacy}","received_events_url":"https://api.github.com/users/lavalamp/received_events","type":"User","site_admin":false},"open_issues":266,"closed_issues":552,"state":"open","created_at":"2016-08-16T21:47:28Z","updated_at":"2016-11-11T17:17:06Z","due_on":"2016-12-09T00:00:00Z","closed_at":null},"comments":11,"created_at":"2016-11-02T14:45:00Z","updated_at":"2016-11-11T17:28:52Z","closed_at":null,"body":"<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.-->\r\n\r\n**Is this a request for help?**: No\r\n\r\n\r\n**What keywords did you search in Kubernetes issues before filing this one?** (If you have found any duplicates, you should instead reply there.): timeout, pd, persistent disk\r\n\r\n---\r\n\r\n**Is this a BUG REPORT or FEATURE REQUEST?** (choose one): BUG REPORT\r\n\r\n<!--\r\nIf this is a BUG REPORT, please:\r\n  - Fill in as much of the template below as you can.  If you leave out\r\n    information, we can't help you as well.\r\n\r\nIf this is a FEATURE REQUEST, please:\r\n  - Describe *in detail* the feature/behavior/change you'd like to see.\r\n\r\nIn both cases, be ready for followup questions, and please respond in a timely\r\nmanner.  If we can't reproduce a bug or think a feature already exists, we\r\nmight close your issue.  If we're wrong, PLEASE feel free to reopen it and\r\nexplain why.\r\n-->\r\n\r\n**Kubernetes version** (use `kubectl version`): \r\n\r\n```\r\nClient Version: version.Info{Major:\"1\", Minor:\"3\", GitVersion:\"v1.3.6\", GitCommit:\"ae4550cc9c89a593bcda6678df201db1b208133b\", GitTreeState:\"clean\", BuildDate:\"2016-08-26T18:13:23Z\", GoVersion:\"go1.6.2\", Compiler:\"gc\", Platform:\"darwin/amd64\"}\r\nServer Version: version.Info{Major:\"1\", Minor:\"4\", GitVersion:\"v1.4.5\", GitCommit:\"5a0a696437ad35c133c0c8493f7e9d22b0f9b81b\", GitTreeState:\"clean\", BuildDate:\"2016-10-29T01:32:42Z\", GoVersion:\"go1.6.3\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\n```\r\n\r\n**Environment**:\r\n- **Cloud provider or hardware configuration**: GCE\r\n- **OS** (e.g. from /etc/os-release):\r\n```\r\nBUILD_ID=8872.26.0\r\nNAME=\"Container-VM Image\"\r\nGOOGLE_CRASH_ID=Lakitu\r\nVERSION_ID=55\r\nBUG_REPORT_URL=https://crbug.com/new\r\nPRETTY_NAME=\"Google Container-VM Image\"\r\nVERSION=55\r\nGOOGLE_METRICS_PRODUCT_ID=26\r\nHOME_URL=\"https://cloud.google.com/compute/docs/containers/vm-image/\"\r\nID=gci\r\n```\r\n\r\n- **Kernel** (e.g. `uname -a`): Linux gke-favega-pool-1-01f44ce6-xr0y 4.4.21+ #1 SMP Wed Oct 26 13:00:52 PDT 2016 x86_64 Intel(R) Xeon(R) CPU @ 2.60GHz GenuineIntel GNU/Linux\r\n- **Install tools**: I don't know what this is\r\n- **Others**: -\r\n\r\n\r\n**What happened**: Early this morning I was woken up because all of our containers with GCE Persistent Disk volume mounts were down. I tried restarting the nodes in the cluster, which worked for all pods but one.\r\n\r\nThe pod in question:\r\n\r\n```\r\nName:\t\tpostgresql-200534733-39m2y\r\nNamespace:\taddressbooks\r\nNode:\t\tgke-favega-pool-1-01f44ce6-xr0y/10.132.0.3\r\nStart Time:\tWed, 02 Nov 2016 14:34:35 +0100\r\nLabels:\t\tapp=addressbooks\r\n\t\tpod-template-hash=200534733\r\n\t\trole=db\r\nStatus:\t\tPending\r\nIP:\t\t\r\nControllers:\tReplicaSet/postgresql-200534733\r\nContainers:\r\n  postgres:\r\n    Container ID:\t\r\n    Image:\t\tpostgres:9.5\r\n    Image ID:\t\t\r\n    Port:\t\t5432/TCP\r\n    State:\t\tWaiting\r\n      Reason:\t\tContainerCreating\r\n    Ready:\t\tFalse\r\n    Restart Count:\t0\r\n    Environment Variables:\r\n      POSTGRES_USER:\t\taddressbooks\r\n      POSTGRES_PASSWORD:\taddressbooks\r\n      PGDATA:\t\t\t/var/lib/postgresql/data/pgdata\r\nConditions:\r\n  Type\t\tStatus\r\n  Initialized \tTrue \r\n  Ready \tFalse \r\n  PodScheduled \tTrue \r\nVolumes:\r\n  postgres-volume:\r\n    Type:\tGCEPersistentDisk (a Persistent Disk resource in Google Compute Engine)\r\n    PDName:\taddressbooks-postgres-volume\r\n    FSType:\text4\r\n    Partition:\t0\r\n    ReadOnly:\tfalse\r\n  default-token-mp5nw:\r\n    Type:\tSecret (a volume populated by a Secret)\r\n    SecretName:\tdefault-token-mp5nw\r\nQoS Tier:\tBestEffort\r\nEvents:\r\n  FirstSeen\tLastSeen\tCount\tFrom\t\t\t\t\t\tSubobjectPath\tType\t\tReason\t\tMessage\r\n  ---------\t--------\t-----\t----\t\t\t\t\t\t-------------\t--------\t------\t\t-------\r\n  1h\t\t1m\t\t29\t{kubelet gke-favega-pool-1-01f44ce6-xr0y}\t\t\tWarning\t\tFailedMount\tUnable to mount volumes for pod \"postgresql-200534733-39m2y_addressbooks(183566c0-a101-11e6-bd04-42010a84016f)\": timeout expired waiting for volumes to attach/mount for pod \"postgresql-200534733-39m2y\"/\"addressbooks\". list of unattached/unmounted volumes=[postgres-volume]\r\n  1h\t\t1m\t\t29\t{kubelet gke-favega-pool-1-01f44ce6-xr0y}\t\t\tWarning\t\tFailedSync\tError syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \"postgresql-200534733-39m2y\"/\"addressbooks\". list of unattached/unmounted volumes=[postgres-volume]\r\n```\r\n\r\nThe deployment for the pod:\r\n\r\n\r\n```\r\napiVersion: extensions/v1beta1\r\nkind: Deployment\r\nmetadata:\r\n  name: postgresql\r\n  namespace: addressbooks\r\nspec:\r\n  replicas: 1\r\n  template:\r\n    metadata:\r\n      labels:\r\n        app: addressbooks\r\n        role: db\r\n    spec:\r\n      containers:\r\n      - name: postgres\r\n        image: postgres:9.5\r\n        ports:\r\n        - containerPort: 5432\r\n        env:\r\n        - name: POSTGRES_USER\r\n          value: addressbooks\r\n        - name: POSTGRES_PASSWORD\r\n          value: addressbooks\r\n        - name: PGDATA\r\n          value: \"/var/lib/postgresql/data/pgdata\"\r\n        volumeMounts:\r\n        - mountPath: \"/var/lib/postgresql/data/\"\r\n          name: postgres-volume\r\n      volumes:\r\n      - name: postgres-volume\r\n        gcePersistentDisk:\r\n          pdName: addressbooks-postgres-volume\r\n          fsType: ext4\r\n```\r\n\r\nNote I can manually mount the PD on the node k8s is trying to schedule the pod on just fine with `gcloud`.\r\n\r\nFishy journalctl:\r\n\r\n```\r\nNov 02 13:55:37 gke-favega-pool-1-01f44ce6-xr0y kubelet[1251]: I1102 13:55:37.185645    1251 conversion.go:134] failed to handle multiple devices for container. Skipping Filesystem stats\r\n\r\nNov 02 13:55:37 gke-favega-pool-1-01f44ce6-xr0y kubelet[1251]: I1102 13:55:37.186721    1251 conversion.go:134] failed to handle multiple devices for container. Skipping Filesystem stats\r\n\r\nNov 02 13:55:41 gke-favega-pool-1-01f44ce6-xr0y kubelet[1251]: I1102 13:55:41.521733    1251 operation_executor.go:803] MountVolume.SetUp succeeded for volume \"kubernetes.io/secret/df702b83-a0f8-11e6-8de3-42010a84016f-default-token-5yom9\" (spec.Name: \"default-token-5yom9\") pod \"df702b83-a0f8-11e6-8de3-42010a84016f\" (UID: \"df702b83-a0f8-11e6-8de3-42010a84016f\").\r\n\r\nNov 02 13:55:41 gke-favega-pool-1-01f44ce6-xr0y kubelet[1251]: I1102 13:55:41.523158    1251 operation_executor.go:803] MountVolume.SetUp succeeded for volume \"kubernetes.io/secret/dfef300a-a0f8-11e6-8de3-42010a84016f-default-token-ck675\" (spec.Name: \"default-token-ck675\") pod \"dfef300a-a0f8-11e6-8de3-42010a84016f\" (UID: \"dfef300a-a0f8-11e6-8de3-42010a84016f\").\r\n\r\nNov 02 13:55:46 gke-favega-pool-1-01f44ce6-xr0y kubelet[1251]: I1102 13:55:46.479252    1251 server.go:967] GET /healthz: (40.214µs) 200 [[curl/7.50.3] 127.0.0.1:51956]\r\n\r\nNov 02 13:55:46 gke-favega-pool-1-01f44ce6-xr0y kubelet[1251]: I1102 13:55:46.551729    1251 operation_executor.go:803] MountVolume.SetUp succeeded for volume \"kubernetes.io/secret/b88fbb51-a0fc-11e6-bd04-42010a84016f-default-token-9qv5p\" (spec.Name: \"default-token-9qv5p\") pod \"b88fbb51-a0fc-11e6-bd04-42010a84016f\" (UID: \"b88fbb51-a0fc-11e6-bd04-42010a84016f\").\r\n\r\nNov 02 13:55:47 gke-favega-pool-1-01f44ce6-xr0y kubelet[1251]: I1102 13:55:47.209007    1251 conversion.go:134] failed to handle multiple devices for container. Skipping Filesystem stats\r\n\r\nNov 02 13:55:47 gke-favega-pool-1-01f44ce6-xr0y kubelet[1251]: I1102 13:55:47.209613    1251 conversion.go:134] failed to handle multiple devices for container. Skipping Filesystem stats\r\n\r\nNov 02 13:55:52 gke-favega-pool-1-01f44ce6-xr0y kubelet[1251]: I1102 13:55:52.527417    1251 operation_executor.go:803] MountVolume.SetUp succeeded for volume \"kubernetes.io/secret/df63b213-a0f8-11e6-8de3-42010a84016f-default-token-slpdg\" (spec.Name: \"default-token-slpdg\") pod \"df63b213-a0f8-11e6-8de3-42010a84016f\" (UID: \"df63b213-a0f8-11e6-8de3-42010a84016f\").\r\n\r\nNov 02 13:55:53 gke-favega-pool-1-01f44ce6-xr0y kubelet[1251]: I1102 13:55:53.518538    1251 operation_executor.go:803] MountVolume.SetUp succeeded for volume \"kubernetes.io/secret/df8a26b2-a0f8-11e6-8de3-42010a84016f-default-token-5yom9\" (spec.Name: \"default-token-5yom9\") pod \"df8a26b2-a0f8-11e6-8de3-42010a84016f\" (UID: \"df8a26b2-a0f8-11e6-8de3-42010a84016f\").\r\n\r\nNov 02 13:55:56 gke-favega-pool-1-01f44ce6-xr0y kubelet[1251]: I1102 13:55:56.501685    1251 server.go:967] GET /healthz: (32.53µs) 200 [[curl/7.50.3] 127.0.0.1:52048]\r\n\r\nNov 02 13:55:57 gke-favega-pool-1-01f44ce6-xr0y kubelet[1251]: I1102 13:55:57.272752    1251 conversion.go:134] failed to handle multiple devices for container. Skipping Filesystem stats\r\n\r\nNov 02 13:55:57 gke-favega-pool-1-01f44ce6-xr0y kubelet[1251]: I1102 13:55:57.273347    1251 conversion.go:134] failed to handle multiple devices for container. Skipping Filesystem stats\r\n\r\nNov 02 13:55:58 gke-favega-pool-1-01f44ce6-xr0y kubelet[1251]: I1102 13:55:58.593168    1251 operation_executor.go:803] MountVolume.SetUp succeeded for volume \"kubernetes.io/secret/dfa7ae50-a0f8-11e6-8de3-42010a84016f-default-token-eobju\" (spec.Name: \"default-token-eobju\") pod \"dfa7ae50-a0f8-11e6-8de3-42010a84016f\" (UID: \"dfa7ae50-a0f8-11e6-8de3-42010a84016f\").\r\n\r\nNov 02 13:56:00 gke-favega-pool-1-01f44ce6-xr0y kubelet[1251]: I1102 13:56:00.601068    1251 operation_executor.go:803] MountVolume.SetUp succeeded for volume \"kubernetes.io/secret/e00fd33c-a0f8-11e6-8de3-42010a84016f-default-token-hwo5r\" (spec.Name: \"default-token-hwo5r\") pod \"e00fd33c-a0f8-11e6-8de3-42010a84016f\" (UID: \"e00fd33c-a0f8-11e6-8de3-42010a84016f\").\r\n\r\nNov 02 13:56:00 gke-favega-pool-1-01f44ce6-xr0y kubelet[1251]: I1102 13:56:00.602974    1251 operation_executor.go:803] MountVolume.SetUp succeeded for volume \"kubernetes.io/secret/e01041a6-a0f8-11e6-8de3-42010a84016f-default-token-hwo5r\" (spec.Name: \"default-token-hwo5r\") pod \"e01041a6-a0f8-11e6-8de3-42010a84016f\" (UID: \"e01041a6-a0f8-11e6-8de3-42010a84016f\").\r\n```\r\n\r\n\r\n**What you expected to happen**: I expected the pod to start just fine, like every other pod in my cluster.\r\n\r\n\r\n**How to reproduce it** (as minimally and precisely as possible): It only fails for this single pod, so I don't know how to reproduce it.\r\n\r\n\r\n**Anything else do we need to know**: I don't think so.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/260009748","html_url":"https://github.com/kubernetes/kubernetes/issues/36067#issuecomment-260009748","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/36067","id":260009748,"user":{"login":"autrilla","id":2260567,"avatar_url":"https://avatars.githubusercontent.com/u/2260567?v=3","gravatar_id":"","url":"https://api.github.com/users/autrilla","html_url":"https://github.com/autrilla","followers_url":"https://api.github.com/users/autrilla/followers","following_url":"https://api.github.com/users/autrilla/following{/other_user}","gists_url":"https://api.github.com/users/autrilla/gists{/gist_id}","starred_url":"https://api.github.com/users/autrilla/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/autrilla/subscriptions","organizations_url":"https://api.github.com/users/autrilla/orgs","repos_url":"https://api.github.com/users/autrilla/repos","events_url":"https://api.github.com/users/autrilla/events{/privacy}","received_events_url":"https://api.github.com/users/autrilla/received_events","type":"User","site_admin":false},"created_at":"2016-11-11T17:28:52Z","updated_at":"2016-11-11T17:28:52Z","body":"It may be worth noting this happened to me with a single node, not several."}}
{"actor":{"display_login":"autrilla"}}