{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/60645","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/60645/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/60645/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/60645/events","html_url":"https://github.com/kubernetes/kubernetes/issues/60645","id":301424451,"number":60645,"title":"Kubelet leaves orphaned directories after downtime","user":{"login":"jsafrane","id":1745006,"avatar_url":"https://avatars2.githubusercontent.com/u/1745006?v=4","gravatar_id":"","url":"https://api.github.com/users/jsafrane","html_url":"https://github.com/jsafrane","followers_url":"https://api.github.com/users/jsafrane/followers","following_url":"https://api.github.com/users/jsafrane/following{/other_user}","gists_url":"https://api.github.com/users/jsafrane/gists{/gist_id}","starred_url":"https://api.github.com/users/jsafrane/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jsafrane/subscriptions","organizations_url":"https://api.github.com/users/jsafrane/orgs","repos_url":"https://api.github.com/users/jsafrane/repos","events_url":"https://api.github.com/users/jsafrane/events{/privacy}","received_events_url":"https://api.github.com/users/jsafrane/received_events","type":"User","site_admin":false},"labels":[{"id":105146071,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/kind/bug","name":"kind/bug","color":"e11d21","default":false},{"id":173493665,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/node","name":"sig/node","color":"d2b48c","default":false},{"id":169428334,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/storage","name":"sig/storage","color":"d2b48c","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-03-01T14:28:54Z","updated_at":"2018-03-01T14:28:55Z","closed_at":null,"author_association":"MEMBER","body":"/kind bug\r\n\r\n**What happened**:\r\nkubelet logs lot of `Orphaned pod \"2663ce08-d054-11e7-b7ee-00505684022a\" found, but volume paths are still present on disk.` when it starts and a pod was deleted while it was offline.\r\n\r\n**How to reproduce it (as minimally and precisely as possible)**:\r\n\r\n1. run a pod with AWS EBS volume (should work also with GCE or any other volume)\r\n2. stop kubelet\r\n3. delete the pod\r\n4. start kubelet\r\n\r\nIt's not always reproducible, is there some sort of race?\r\n\r\nMy current theory:\r\n\r\n* desiredStateOfWorldPopulator scans pods and adds this pod to DSW\r\n\r\n* reconciler.syncStates() runs.\r\n  * it finds mounted volume\r\n  * but it does **not** add it to actual state of world because it's in DSW:\r\nhttps://github.com/openshift/origin/blob/master/vendor/k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler/reconciler.go#L366\r\n\r\n* When kubelet fully starts, it kills the pod and cleans DSW, but the volume never reached AWS so nothing is unmounted. \r\n\r\n\r\n**Environment**:\r\n- Kubernetes version (use `kubectl version`): today's master\r\n\r\nI have troubles reproducing this on master.\r\n\r\n/sig storage\r\n/sig node\r\n@jingxu97 @gnufied "},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/369608718","html_url":"https://github.com/kubernetes/kubernetes/issues/60645#issuecomment-369608718","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/60645","id":369608718,"user":{"login":"k8s-ci-robot","id":20407524,"avatar_url":"https://avatars0.githubusercontent.com/u/20407524?v=4","gravatar_id":"","url":"https://api.github.com/users/k8s-ci-robot","html_url":"https://github.com/k8s-ci-robot","followers_url":"https://api.github.com/users/k8s-ci-robot/followers","following_url":"https://api.github.com/users/k8s-ci-robot/following{/other_user}","gists_url":"https://api.github.com/users/k8s-ci-robot/gists{/gist_id}","starred_url":"https://api.github.com/users/k8s-ci-robot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/k8s-ci-robot/subscriptions","organizations_url":"https://api.github.com/users/k8s-ci-robot/orgs","repos_url":"https://api.github.com/users/k8s-ci-robot/repos","events_url":"https://api.github.com/users/k8s-ci-robot/events{/privacy}","received_events_url":"https://api.github.com/users/k8s-ci-robot/received_events","type":"User","site_admin":false},"created_at":"2018-03-01T14:28:55Z","updated_at":"2018-03-01T14:28:55Z","author_association":"COLLABORATOR","body":"@jsafrane: There are no sig labels on this issue. Please add a sig label.\n\n<details>\n\nA sig label can be added by either:\n\n1. mentioning a sig: `@kubernetes/sig-<group-name>-<group-suffix>`\n    e.g., `@kubernetes/sig-contributor-experience-<group-suffix>` to notify the contributor experience sig, OR\n\n2. specifying the label manually: `/sig <group-name>`\n    e.g., `/sig scalability` to apply the `sig/scalability` label\n\nNote: Method 1 will trigger an email to the group. See the [group list](https://git.k8s.io/community/sig-list.md).\nThe `<group-suffix>` in method 1 has to be replaced with one of these: _**bugs, feature-requests, pr-reviews, test-failures, proposals**_\n\nInstructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/devel/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.\n</details>"}}
{"actor":{"display_login":"k8s-ci-robot"}}