{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/70319","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/70319/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/70319/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/70319/events","html_url":"https://github.com/kubernetes/kubernetes/issues/70319","id":374587351,"node_id":"MDU6SXNzdWUzNzQ1ODczNTE=","number":70319,"title":"Kubelet VolumeManager incorrectly marks volume as not \"in use\" (i.e. safe to detach) while a mount or format is pending","user":{"login":"saad-ali","id":10052848,"node_id":"MDQ6VXNlcjEwMDUyODQ4","avatar_url":"https://avatars1.githubusercontent.com/u/10052848?v=4","gravatar_id":"","url":"https://api.github.com/users/saad-ali","html_url":"https://github.com/saad-ali","followers_url":"https://api.github.com/users/saad-ali/followers","following_url":"https://api.github.com/users/saad-ali/following{/other_user}","gists_url":"https://api.github.com/users/saad-ali/gists{/gist_id}","starred_url":"https://api.github.com/users/saad-ali/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/saad-ali/subscriptions","organizations_url":"https://api.github.com/users/saad-ali/orgs","repos_url":"https://api.github.com/users/saad-ali/repos","events_url":"https://api.github.com/users/saad-ali/events{/privacy}","received_events_url":"https://api.github.com/users/saad-ali/received_events","type":"User","site_admin":false},"labels":[{"id":105146071,"node_id":"MDU6TGFiZWwxMDUxNDYwNzE=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/kind/bug","name":"kind/bug","color":"e11d21","default":false},{"id":114528068,"node_id":"MDU6TGFiZWwxMTQ1MjgwNjg=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/priority/critical-urgent","name":"priority/critical-urgent","color":"e11d21","default":false},{"id":169428334,"node_id":"MDU6TGFiZWwxNjk0MjgzMzQ=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/storage","name":"sig/storage","color":"d2b48c","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/40","html_url":"https://github.com/kubernetes/kubernetes/milestone/40","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/40/labels","id":2913713,"node_id":"MDk6TWlsZXN0b25lMjkxMzcxMw==","number":40,"title":"v1.13","description":"","creator":{"login":"thockin","id":5595220,"node_id":"MDQ6VXNlcjU1OTUyMjA=","avatar_url":"https://avatars0.githubusercontent.com/u/5595220?v=4","gravatar_id":"","url":"https://api.github.com/users/thockin","html_url":"https://github.com/thockin","followers_url":"https://api.github.com/users/thockin/followers","following_url":"https://api.github.com/users/thockin/following{/other_user}","gists_url":"https://api.github.com/users/thockin/gists{/gist_id}","starred_url":"https://api.github.com/users/thockin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thockin/subscriptions","organizations_url":"https://api.github.com/users/thockin/orgs","repos_url":"https://api.github.com/users/thockin/repos","events_url":"https://api.github.com/users/thockin/events{/privacy}","received_events_url":"https://api.github.com/users/thockin/received_events","type":"User","site_admin":false},"open_issues":76,"closed_issues":55,"state":"open","created_at":"2017-11-15T06:32:59Z","updated_at":"2018-10-27T00:32:27Z","due_on":null,"closed_at":null},"comments":2,"created_at":"2018-10-27T00:32:08Z","updated_at":"2018-10-27T09:18:19Z","closed_at":null,"author_association":"MEMBER","body":"<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!-->\r\n\r\n\r\n**What happened**:\r\nWhile debugging an issue with a volume getting detached prematurely, @jingxu97 discovered an issue: the Kubelet VolumeManager incorrectly marks volume as not \"in use\" (i.e. safe to detach) while a mount or format is pending.\r\n\r\nFor context, attach/detach controller will not proceed with a detach (after pod referencing a volume is deleted) while the volume is still marked as \"in use\" by kubelet. Kubelet marks a volume as \"in use\" before it starts mounting or formating the volume.\r\n\r\nThe problem is:\r\n\r\n* If a pod referencing a volume is deleted while a mount (or format) is pending, then kubelet will go ahead and remove that pod from its \"desired state of the world\" cache (normally this waits for pod's containers to terminate, but in this case the containers never even started because mount is still pending, i.e. format, is still pending).\r\n* Kubelet volume_manager's `GetVolumesInUse()` method (https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/volumemanager/volume_manager.go#L293) reports the volume as no longer in use because it is not in desired or actual state (it does not check pending operations <-- this is the bug).\r\n* Attach/detach controller sees the pod delete, checks and sees the volume is not marked as \"in use\" and proceeds with the detach (incorrectly), but a mount (or format) operation may be pending.\r\n\r\n**What you expected to happen**:\r\n\r\nKubelet volume_manager's `GetVolumesInUse()` method (https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/volumemanager/volume_manager.go#L293) should count volumes that have pending operations (mount or otherwise) as \"in use\" (even if they are not in actual or desired state).\r\n\r\n**How to reproduce it (as minimally and precisely as possible)**:\r\n\r\nWhile mount is pending (because format is taking a long time, for example), delete the pod referencing the attachable volume.\r\n\r\n**Anything else we need to know?**:\r\n/sig storage\r\n/milestone 1.13\r\n\r\n**Environment**:\r\n- Kubernetes version (use `kubectl version`):\r\n- Cloud provider or hardware configuration:\r\n- OS (e.g. from /etc/os-release):\r\n- Kernel (e.g. `uname -a`):\r\n- Install tools:\r\n- Others:\r\n\r\n<!-- DO NOT EDIT BELOW THIS LINE -->\r\n/kind bug"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/433604856","html_url":"https://github.com/kubernetes/kubernetes/issues/70319#issuecomment-433604856","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/70319","id":433604856,"node_id":"MDEyOklzc3VlQ29tbWVudDQzMzYwNDg1Ng==","user":{"login":"mlmhl","id":7688266,"node_id":"MDQ6VXNlcjc2ODgyNjY=","avatar_url":"https://avatars2.githubusercontent.com/u/7688266?v=4","gravatar_id":"","url":"https://api.github.com/users/mlmhl","html_url":"https://github.com/mlmhl","followers_url":"https://api.github.com/users/mlmhl/followers","following_url":"https://api.github.com/users/mlmhl/following{/other_user}","gists_url":"https://api.github.com/users/mlmhl/gists{/gist_id}","starred_url":"https://api.github.com/users/mlmhl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mlmhl/subscriptions","organizations_url":"https://api.github.com/users/mlmhl/orgs","repos_url":"https://api.github.com/users/mlmhl/repos","events_url":"https://api.github.com/users/mlmhl/events{/privacy}","received_events_url":"https://api.github.com/users/mlmhl/received_events","type":"User","site_admin":false},"created_at":"2018-10-27T09:18:19Z","updated_at":"2018-10-27T09:18:19Z","author_association":"CONTRIBUTOR","body":"It seems that we need to mark all attached volumes in `ActualStateOfWorld` as `in-use`, not only globally mounted volumes.\r\n\r\nWe can add a `GetInUseVolumes` method to ASW and return all volumes exist in `ActualStateOfWorld.attachedVolumes`."}}
{"actor":{"display_login":"mlmhl"}}