{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/29324","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/29324/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/29324/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/29324/events","html_url":"https://github.com/kubernetes/kubernetes/issues/29324","id":166686746,"number":29324,"title":"Wrong AWS volume can be mounted","user":{"login":"atombender","id":50314,"avatar_url":"https://avatars.githubusercontent.com/u/50314?v=3","gravatar_id":"","url":"https://api.github.com/users/atombender","html_url":"https://github.com/atombender","followers_url":"https://api.github.com/users/atombender/followers","following_url":"https://api.github.com/users/atombender/following{/other_user}","gists_url":"https://api.github.com/users/atombender/gists{/gist_id}","starred_url":"https://api.github.com/users/atombender/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/atombender/subscriptions","organizations_url":"https://api.github.com/users/atombender/orgs","repos_url":"https://api.github.com/users/atombender/repos","events_url":"https://api.github.com/users/atombender/events{/privacy}","received_events_url":"https://api.github.com/users/atombender/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/area/platform/aws","name":"area/platform/aws","color":"d4c5f9"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/area/storage","name":"area/storage","color":"006b75"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/priority/P0","name":"priority/P0","color":"e11d21"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/cluster","name":"team/cluster","color":"d2b48c"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/sig-aws","name":"team/sig-aws","color":"d2b48c"}],"state":"open","locked":false,"assignee":{"login":"justinsb","id":100893,"avatar_url":"https://avatars.githubusercontent.com/u/100893?v=3","gravatar_id":"","url":"https://api.github.com/users/justinsb","html_url":"https://github.com/justinsb","followers_url":"https://api.github.com/users/justinsb/followers","following_url":"https://api.github.com/users/justinsb/following{/other_user}","gists_url":"https://api.github.com/users/justinsb/gists{/gist_id}","starred_url":"https://api.github.com/users/justinsb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/justinsb/subscriptions","organizations_url":"https://api.github.com/users/justinsb/orgs","repos_url":"https://api.github.com/users/justinsb/repos","events_url":"https://api.github.com/users/justinsb/events{/privacy}","received_events_url":"https://api.github.com/users/justinsb/received_events","type":"User","site_admin":false},"assignees":[{"login":"justinsb","id":100893,"avatar_url":"https://avatars.githubusercontent.com/u/100893?v=3","gravatar_id":"","url":"https://api.github.com/users/justinsb","html_url":"https://github.com/justinsb","followers_url":"https://api.github.com/users/justinsb/followers","following_url":"https://api.github.com/users/justinsb/following{/other_user}","gists_url":"https://api.github.com/users/justinsb/gists{/gist_id}","starred_url":"https://api.github.com/users/justinsb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/justinsb/subscriptions","organizations_url":"https://api.github.com/users/justinsb/orgs","repos_url":"https://api.github.com/users/justinsb/repos","events_url":"https://api.github.com/users/justinsb/events{/privacy}","received_events_url":"https://api.github.com/users/justinsb/received_events","type":"User","site_admin":false},{"login":"jingxu97","id":19172805,"avatar_url":"https://avatars.githubusercontent.com/u/19172805?v=3","gravatar_id":"","url":"https://api.github.com/users/jingxu97","html_url":"https://github.com/jingxu97","followers_url":"https://api.github.com/users/jingxu97/followers","following_url":"https://api.github.com/users/jingxu97/following{/other_user}","gists_url":"https://api.github.com/users/jingxu97/gists{/gist_id}","starred_url":"https://api.github.com/users/jingxu97/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jingxu97/subscriptions","organizations_url":"https://api.github.com/users/jingxu97/orgs","repos_url":"https://api.github.com/users/jingxu97/repos","events_url":"https://api.github.com/users/jingxu97/events{/privacy}","received_events_url":"https://api.github.com/users/jingxu97/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/18","html_url":"https://github.com/kubernetes/kubernetes/milestones/v1.3","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/18/labels","id":1638460,"number":18,"title":"v1.3","description":"https://github.com/kubernetes/features/blob/master/release-1.3/release-1.3.md","creator":{"login":"bgrant0607","id":7725777,"avatar_url":"https://avatars.githubusercontent.com/u/7725777?v=3","gravatar_id":"","url":"https://api.github.com/users/bgrant0607","html_url":"https://github.com/bgrant0607","followers_url":"https://api.github.com/users/bgrant0607/followers","following_url":"https://api.github.com/users/bgrant0607/following{/other_user}","gists_url":"https://api.github.com/users/bgrant0607/gists{/gist_id}","starred_url":"https://api.github.com/users/bgrant0607/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bgrant0607/subscriptions","organizations_url":"https://api.github.com/users/bgrant0607/orgs","repos_url":"https://api.github.com/users/bgrant0607/repos","events_url":"https://api.github.com/users/bgrant0607/events{/privacy}","received_events_url":"https://api.github.com/users/bgrant0607/received_events","type":"User","site_admin":false},"open_issues":14,"closed_issues":1242,"state":"open","created_at":"2016-03-11T07:40:20Z","updated_at":"2016-09-26T19:47:25Z","due_on":"2016-06-24T00:00:00Z","closed_at":null},"comments":79,"created_at":"2016-07-20T21:29:29Z","updated_at":"2016-10-04T18:09:58Z","closed_at":null,"body":"Background: K8s often has problems attaching/detaching AWS volumes: If I do a `kubectl replace` on a deployment, I frequently observe K8s not correctly detaching the volume, and the new deployment will be forever stuck in a crash loop because it times out getting the new volume. Seems like a race condition. The steps I always do when this happens is to delete the deployment, manually force-detach via EC2 API, and manually `umount` the mounts, otherwise they never seem to be cleaned up.\r\n\r\nSo today I experienced this again, and I did the following:\r\n\r\n1. Delete deployment.\r\n2. Manually detach volume via EC2 API, since it was not being released.\r\n3. **Manually `umount` wrong volume**. I accidentally unmounted the volume of a different pod because I didn't read the `mount` output correctly. Oops.\r\n4. Created deployment.\r\n5. Noticed my mistake.\r\n\r\nWhat happened was that the new deployment was mounted with the wrong device:\r\n\r\n```shell\r\n$ mount | grep xvdb\r\n/dev/xvdba on /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/vol-0e3bf8343563ee5aa type ext4 (rw,relatime,data=ordered)\r\n/dev/xvdba on /var/lib/kubelet/pods/a8f54bf6-4ebd-11e6-89fc-065b459e106f/volumes/kubernetes.io~aws-ebs/data type ext4 (rw,relatime,data=ordered)\r\n```\r\n\r\nThis is the right volume _ID_, but it has mounted the wrong device. According to EC2, the (right) volume is attached to `/dev/xvdbb`.\r\n\r\n```shell\r\n$ aws ec2 describe-volumes --volume-ids vol-0e3bf8343563ee5aa\r\n{\r\n    \"Volumes\": [\r\n        {\r\n            \"AvailabilityZone\": \"us-east-1e\", \r\n            \"Attachments\": [\r\n                {\r\n                    \"AttachTime\": \"2016-07-20T21:05:17.000Z\", \r\n                    \"InstanceId\": \"i-06d03923e9e5a6329\", \r\n                    \"VolumeId\": \"vol-0e3bf8343563ee5aa\", \r\n                    \"State\": \"attached\", \r\n                    \"DeleteOnTermination\": false, \r\n                    \"Device\": \"/dev/xvdbb\"\r\n                }\r\n            ], \r\n            \"Tags\": [],\r\n            \"Encrypted\": false, \r\n            \"VolumeType\": \"gp2\", \r\n            \"VolumeId\": \"vol-0e3bf8343563ee5aa\", \r\n            \"State\": \"in-use\", \r\n            \"Iops\": 100, \r\n            \"SnapshotId\": \"\", \r\n            \"CreateTime\": \"2016-07-19T00:25:00.168Z\", \r\n            \"Size\": 30\r\n        }\r\n    ]\r\n}\r\n```\r\n\r\n`/dev/xvdba` is the volume I accidentally unmounted by mistake:\r\n\r\n```javascript\r\n            // ...\r\n            \"Attachments\": [\r\n                {\r\n                    \"AttachTime\": \"2016-07-19T17:56:27.000Z\", \r\n                    \"InstanceId\": \"i-06d03923e9e5a6329\", \r\n                    \"VolumeId\": \"vol-0e0a13f1b3a4a6dbc\", \r\n                    \"State\": \"attached\", \r\n                    \"DeleteOnTermination\": false, \r\n                    \"Device\": \"/dev/xvdba\"\r\n                }\r\n            ], \r\n```\r\n\r\nSo it's mounting the wrong volume into the pod.\r\n\r\nOf course I was being naughty, but it seems to me that this should never happen, since EC2 has already allocated `xvdbb`?\r\n\r\n[Here's](https://gist.github.com/atombender/a1c8199ed202c2ea0d6b4566db65c60b) the entire KCM output from this fiasco."},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/251467410","html_url":"https://github.com/kubernetes/kubernetes/issues/29324#issuecomment-251467410","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/29324","id":251467410,"user":{"login":"argusua","id":4500733,"avatar_url":"https://avatars.githubusercontent.com/u/4500733?v=3","gravatar_id":"","url":"https://api.github.com/users/argusua","html_url":"https://github.com/argusua","followers_url":"https://api.github.com/users/argusua/followers","following_url":"https://api.github.com/users/argusua/following{/other_user}","gists_url":"https://api.github.com/users/argusua/gists{/gist_id}","starred_url":"https://api.github.com/users/argusua/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/argusua/subscriptions","organizations_url":"https://api.github.com/users/argusua/orgs","repos_url":"https://api.github.com/users/argusua/repos","events_url":"https://api.github.com/users/argusua/events{/privacy}","received_events_url":"https://api.github.com/users/argusua/received_events","type":"User","site_admin":false},"created_at":"2016-10-04T18:09:58Z","updated_at":"2016-10-04T18:09:58Z","body":"Thank you, @jingxu97 \r\nUnfortunately I have a vacation and I'll come back only after 2 weeks. I hope somebody can help with testing also."}}
{"actor":{"display_login":"argusua"}}