{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/65246","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/65246/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/65246/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/65246/events","html_url":"https://github.com/kubernetes/kubernetes/issues/65246","id":333860191,"node_id":"MDU6SXNzdWUzMzM4NjAxOTE=","number":65246,"title":"Second Pod Using Same PVC Fails WaitForAttach Flakily","user":{"login":"davidz627","id":5459214,"node_id":"MDQ6VXNlcjU0NTkyMTQ=","avatar_url":"https://avatars3.githubusercontent.com/u/5459214?v=4","gravatar_id":"","url":"https://api.github.com/users/davidz627","html_url":"https://github.com/davidz627","followers_url":"https://api.github.com/users/davidz627/followers","following_url":"https://api.github.com/users/davidz627/following{/other_user}","gists_url":"https://api.github.com/users/davidz627/gists{/gist_id}","starred_url":"https://api.github.com/users/davidz627/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidz627/subscriptions","organizations_url":"https://api.github.com/users/davidz627/orgs","repos_url":"https://api.github.com/users/davidz627/repos","events_url":"https://api.github.com/users/davidz627/events{/privacy}","received_events_url":"https://api.github.com/users/davidz627/received_events","type":"User","site_admin":false},"labels":[{"id":105146071,"node_id":"MDU6TGFiZWwxMDUxNDYwNzE=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/kind/bug","name":"kind/bug","color":"e11d21","default":false},{"id":264749912,"node_id":"MDU6TGFiZWwyNjQ3NDk5MTI=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/kind/flake","name":"kind/flake","color":"f7c6c7","default":false},{"id":114528223,"node_id":"MDU6TGFiZWwxMTQ1MjgyMjM=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/priority/important-soon","name":"priority/important-soon","color":"eb6420","default":false},{"id":169428334,"node_id":"MDU6TGFiZWwxNjk0MjgzMzQ=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/storage","name":"sig/storage","color":"d2b48c","default":false}],"state":"open","locked":false,"assignee":{"login":"gnufied","id":278,"node_id":"MDQ6VXNlcjI3OA==","avatar_url":"https://avatars0.githubusercontent.com/u/278?v=4","gravatar_id":"","url":"https://api.github.com/users/gnufied","html_url":"https://github.com/gnufied","followers_url":"https://api.github.com/users/gnufied/followers","following_url":"https://api.github.com/users/gnufied/following{/other_user}","gists_url":"https://api.github.com/users/gnufied/gists{/gist_id}","starred_url":"https://api.github.com/users/gnufied/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gnufied/subscriptions","organizations_url":"https://api.github.com/users/gnufied/orgs","repos_url":"https://api.github.com/users/gnufied/repos","events_url":"https://api.github.com/users/gnufied/events{/privacy}","received_events_url":"https://api.github.com/users/gnufied/received_events","type":"User","site_admin":false},"assignees":[{"login":"gnufied","id":278,"node_id":"MDQ6VXNlcjI3OA==","avatar_url":"https://avatars0.githubusercontent.com/u/278?v=4","gravatar_id":"","url":"https://api.github.com/users/gnufied","html_url":"https://github.com/gnufied","followers_url":"https://api.github.com/users/gnufied/followers","following_url":"https://api.github.com/users/gnufied/following{/other_user}","gists_url":"https://api.github.com/users/gnufied/gists{/gist_id}","starred_url":"https://api.github.com/users/gnufied/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gnufied/subscriptions","organizations_url":"https://api.github.com/users/gnufied/orgs","repos_url":"https://api.github.com/users/gnufied/repos","events_url":"https://api.github.com/users/gnufied/events{/privacy}","received_events_url":"https://api.github.com/users/gnufied/received_events","type":"User","site_admin":false},{"login":"pohly","id":22076,"node_id":"MDQ6VXNlcjIyMDc2","avatar_url":"https://avatars1.githubusercontent.com/u/22076?v=4","gravatar_id":"","url":"https://api.github.com/users/pohly","html_url":"https://github.com/pohly","followers_url":"https://api.github.com/users/pohly/followers","following_url":"https://api.github.com/users/pohly/following{/other_user}","gists_url":"https://api.github.com/users/pohly/gists{/gist_id}","starred_url":"https://api.github.com/users/pohly/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pohly/subscriptions","organizations_url":"https://api.github.com/users/pohly/orgs","repos_url":"https://api.github.com/users/pohly/repos","events_url":"https://api.github.com/users/pohly/events{/privacy}","received_events_url":"https://api.github.com/users/pohly/received_events","type":"User","site_admin":false},{"login":"saad-ali","id":10052848,"node_id":"MDQ6VXNlcjEwMDUyODQ4","avatar_url":"https://avatars1.githubusercontent.com/u/10052848?v=4","gravatar_id":"","url":"https://api.github.com/users/saad-ali","html_url":"https://github.com/saad-ali","followers_url":"https://api.github.com/users/saad-ali/followers","following_url":"https://api.github.com/users/saad-ali/following{/other_user}","gists_url":"https://api.github.com/users/saad-ali/gists{/gist_id}","starred_url":"https://api.github.com/users/saad-ali/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/saad-ali/subscriptions","organizations_url":"https://api.github.com/users/saad-ali/orgs","repos_url":"https://api.github.com/users/saad-ali/repos","events_url":"https://api.github.com/users/saad-ali/events{/privacy}","received_events_url":"https://api.github.com/users/saad-ali/received_events","type":"User","site_admin":false},{"login":"msau42","id":24448061,"node_id":"MDQ6VXNlcjI0NDQ4MDYx","avatar_url":"https://avatars3.githubusercontent.com/u/24448061?v=4","gravatar_id":"","url":"https://api.github.com/users/msau42","html_url":"https://github.com/msau42","followers_url":"https://api.github.com/users/msau42/followers","following_url":"https://api.github.com/users/msau42/following{/other_user}","gists_url":"https://api.github.com/users/msau42/gists{/gist_id}","starred_url":"https://api.github.com/users/msau42/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/msau42/subscriptions","organizations_url":"https://api.github.com/users/msau42/orgs","repos_url":"https://api.github.com/users/msau42/repos","events_url":"https://api.github.com/users/msau42/events{/privacy}","received_events_url":"https://api.github.com/users/msau42/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/40","html_url":"https://github.com/kubernetes/kubernetes/milestone/40","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/40/labels","id":2913713,"node_id":"MDk6TWlsZXN0b25lMjkxMzcxMw==","number":40,"title":"v1.13","description":"","creator":{"login":"thockin","id":5595220,"node_id":"MDQ6VXNlcjU1OTUyMjA=","avatar_url":"https://avatars0.githubusercontent.com/u/5595220?v=4","gravatar_id":"","url":"https://api.github.com/users/thockin","html_url":"https://github.com/thockin","followers_url":"https://api.github.com/users/thockin/followers","following_url":"https://api.github.com/users/thockin/following{/other_user}","gists_url":"https://api.github.com/users/thockin/gists{/gist_id}","starred_url":"https://api.github.com/users/thockin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thockin/subscriptions","organizations_url":"https://api.github.com/users/thockin/orgs","repos_url":"https://api.github.com/users/thockin/repos","events_url":"https://api.github.com/users/thockin/events{/privacy}","received_events_url":"https://api.github.com/users/thockin/received_events","type":"User","site_admin":false},"open_issues":77,"closed_issues":199,"state":"open","created_at":"2017-11-15T06:32:59Z","updated_at":"2018-11-15T13:04:58Z","due_on":null,"closed_at":null},"comments":25,"created_at":"2018-06-19T22:22:46Z","updated_at":"2018-11-15T13:30:43Z","closed_at":null,"author_association":"COLLABORATOR","body":"The test `[sig-storage] CSI Volumes CSI plugin test using CSI driver: hostPath should provision storage` flakes very occasionally. Below are the findings of the circumstances of the flake.\r\n\r\nDuring `testDynamicProvisioning` two pods are created in quick succession. The test flakes when the second Pod fails to come up.\r\n\r\nThe second pod fails to come up with error:\r\n`MountVolume.WaitForAttach failed for volume \"pvc-bc2294b873ef11e8\" : resource name may not be empty`\r\n\r\nUpon exploring the logs we discovered that there was only 1 Attach call made at the beginning, and one Detach call upon timeout of the test [See timestamps]:\r\n```\r\nI0613 19:00:41.888473       1 reconciler.go:291] attacherDetacher.AttachVolume started for volume \"pvc-0c896e426f3c11e8\" (UniqueName: \"kubernetes.io/csi/csi-hostpath^0da20430-6f3c-11e8-a80a-0a580a40016c\") from node \"e2e-40645-674b9-minion-group-gz10\" \r\nI0613 19:00:42.533343       1 operation_generator.go:335] AttachVolume.Attach succeeded for volume \"pvc-0c896e426f3c11e8\" (UniqueName: \"kubernetes.io/csi/csi-hostpath^0da20430-6f3c-11e8-a80a-0a580a40016c\") from node \"e2e-40645-674b9-minion-group-gz10\" \r\nI0613 19:15:52.528902       1 reconciler.go:233] attacherDetacher.DetachVolume started for volume \"pvc-0c896e426f3c11e8\" (UniqueName: \"kubernetes.io/csi/csi-hostpath^0da20430-6f3c-11e8-a80a-0a580a40016c\") on node \"e2e-40645-674b9-minion-group-gz10\" \r\nI0613 19:15:52.615262       1 operation_generator.go:1225] Verified volume is safe to detach for volume \"pvc-0c896e426f3c11e8\" (UniqueName: \"kubernetes.io/csi/csi-hostpath^0da20430-6f3c-11e8-a80a-0a580a40016c\") on node \"e2e-40645-674b9-minion-group-gz10\" \r\nI0613 19:15:53.367487       1 operation_generator.go:419] DetachVolume.Detach succeeded for volume \"pvc-0c896e426f3c11e8\" (UniqueName: \"kubernetes.io/csi/csi-hostpath^0da20430-6f3c-11e8-a80a-0a580a40016c\") on node \"e2e-40645-674b9-minion-group-gz10\" \r\n```\r\n\r\nWe tracked down the `resource name may not be empty` error to the likely culprit in func `waitForVolumeAttachmentInternal`:\r\n```\r\nattach, err := c.k8s.StorageV1beta1().VolumeAttachments().Get(attachID, meta.GetOptions{})\r\n```\r\nbased on the error message returned. \r\n\r\nThis is likely caused by `attachID`(string) being empty [uncomfirmed], which is what the Kubelet would get from the DevicePath field on the Node object. This DevicePath field is normally added on Attach and removed on Detach of the volume.\r\n\r\nThe DevicePath is clearly present on the Node when the first pod is being brought up. But it is possible that it is somehow missing or not able to be retrieved by the time the second Pod is starting up (and continues to be unavailable exhausting the 15m timeout on the test) even though no Detach operation was performed on the volume in the meantime (according to logs). We are currently unable to reproduce this flake on command and did all analysis based on logs from the two flakes we saw naturally occur in the 100s of runs performed of the test.\r\n\r\nThis flake was originally seen in #65064 and the test logs can be found here: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/65064/pull-kubernetes-e2e-gce/40645/\r\n\r\n**Environment**: GCE\r\n- Kubernetes version (use `kubectl version`):\r\n```\r\nClient Version: version.Info{Major:\"1\", Minor:\"8\", GitVersion:\"v1.8.6\", GitCommit:\"6260bb08c46c31eea6cb538b34a9ceb3e406689c\", GitTreeState:\"clean\", BuildDate:\"2017-12-21T06:34:11Z\", GoVersion:\"go1.8.3\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\nServer Version: version.Info{Major:\"1\", Minor:\"12+\", GitVersion:\"v1.12.0-alpha.0.975+98194b8630958e\", GitCommit:\"98194b8630958ee48db662f42404dc0bc6a2ed80\", GitTreeState:\"clean\", BuildDate:\"2018-06-19T18:16:46Z\", GoVersion:\"go1.10.2\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\n```\r\n- Cloud provider or hardware configuration: GCP\r\n- OS (e.g. from /etc/os-release):\r\n```\r\nBUILD_ID=10323.64.0\r\nNAME=\"Container-Optimized OS\"\r\nKERNEL_COMMIT_ID=94bdaa2af49562e7aba0168cae050911d105f5cb\r\nGOOGLE_CRASH_ID=Lakitu\r\nVERSION_ID=65\r\nBUG_REPORT_URL=https://crbug.com/new\r\nPRETTY_NAME=\"Container-Optimized OS from Google\"\r\nVERSION=65\r\nGOOGLE_METRICS_PRODUCT_ID=26\r\nHOME_URL=\"https://cloud.google.com/compute/docs/containers/vm-image/\"\r\nID=cos\r\n```\r\n- Kernel (e.g. `uname -a`): `Linux e2e-test-dyzz-master 4.4.111+ #1 SMP Wed Mar 28 13:44:38 PDT 2018 x86_64 Intel(R) Xeon(R) CPU @ 2.30GHz GenuineIntel GNU/Linux`\r\n\r\n/kind flake\r\n/kind bug\r\n/sig storage\r\n/cc @msau42 @jingxu97 @saad-ali @verult "},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/439040487","html_url":"https://github.com/kubernetes/kubernetes/issues/65246#issuecomment-439040487","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/65246","id":439040487,"node_id":"MDEyOklzc3VlQ29tbWVudDQzOTA0MDQ4Nw==","user":{"login":"jsafrane","id":1745006,"node_id":"MDQ6VXNlcjE3NDUwMDY=","avatar_url":"https://avatars2.githubusercontent.com/u/1745006?v=4","gravatar_id":"","url":"https://api.github.com/users/jsafrane","html_url":"https://github.com/jsafrane","followers_url":"https://api.github.com/users/jsafrane/followers","following_url":"https://api.github.com/users/jsafrane/following{/other_user}","gists_url":"https://api.github.com/users/jsafrane/gists{/gist_id}","starred_url":"https://api.github.com/users/jsafrane/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jsafrane/subscriptions","organizations_url":"https://api.github.com/users/jsafrane/orgs","repos_url":"https://api.github.com/users/jsafrane/repos","events_url":"https://api.github.com/users/jsafrane/events{/privacy}","received_events_url":"https://api.github.com/users/jsafrane/received_events","type":"User","site_admin":false},"created_at":"2018-11-15T13:30:43Z","updated_at":"2018-11-15T13:30:43Z","author_association":"MEMBER","body":"Fix: https://github.com/kubernetes/kubernetes/pull/71074\r\n\r\nAny idea how to test it for real? Fake CSI plugin with slow NodeUnstage would be the best, however we don't have one."}}
{"actor":{"display_login":"jsafrane"}}