{"action":"opened","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/31596","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/31596/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/31596/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/31596/events","html_url":"https://github.com/kubernetes/kubernetes/issues/31596","id":173710634,"number":31596,"title":"Error during cleaning up orphaned pods","user":{"login":"maciaszczykm","id":2823399,"avatar_url":"https://avatars.githubusercontent.com/u/2823399?v=3","gravatar_id":"","url":"https://api.github.com/users/maciaszczykm","html_url":"https://github.com/maciaszczykm","followers_url":"https://api.github.com/users/maciaszczykm/followers","following_url":"https://api.github.com/users/maciaszczykm/following{/other_user}","gists_url":"https://api.github.com/users/maciaszczykm/gists{/gist_id}","starred_url":"https://api.github.com/users/maciaszczykm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/maciaszczykm/subscriptions","organizations_url":"https://api.github.com/users/maciaszczykm/orgs","repos_url":"https://api.github.com/users/maciaszczykm/repos","events_url":"https://api.github.com/users/maciaszczykm/events{/privacy}","received_events_url":"https://api.github.com/users/maciaszczykm/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-08-29T07:12:37Z","updated_at":"2016-08-29T07:12:37Z","closed_at":null,"body":"Following discussion in https://github.com/kubernetes/kube-deploy/issues/192 I'm creating this issue to clear some things and to work on a solution for possible issue:\r\n\r\n> @luxas Okay, it seems, that me and @floreks have found out what causes the issue with cleaning up orphaned pods.\r\n\r\n>On newest master version we're no longer able to reproduce it using steps, that I've previously described, but the issue still occurs. Simplest way to reproduce it is to clean and restart `etcd` and then re-run `kubelet`. When `kubelet` is starting it tries to cleanup orphaned pod directories:\r\n\r\n>`pkg/kubelet/kubelet.go:1708 [master]`\r\n``` go\r\n// cleanupOrphanedPodDirs removes the volumes of pods that should not be\r\n// running and that have no containers running.\r\nfunc (kl *Kubelet) cleanupOrphanedPodDirs(\r\n\r\n>    // ...\r\n\r\n>\tfound, err := kl.listPodsFromDisk()\r\n\tif err != nil {\r\n\t\treturn err\r\n\t}\r\n\terrlist := []error{}\r\n\tfor _, uid := range found {\r\n\t\tif allPods.Has(string(uid)) {\r\n\t\t\tcontinue\r\n\t\t}\r\n\t\tif podVolumesExist := kl.podVolumesExist(uid); podVolumesExist {\r\n\t\t\t// If volumes have not been unmounted/detached, do not delete directory.\r\n\t\t\t// Doing so may result in corruption of data.\r\n\t\t\tglog.V(3).Infof(\"Orphaned pod %q found, but volumes are not cleaned up; err: %v\", uid, err)\r\n\t\t\tcontinue\r\n\t\t}\r\n\t\tglog.V(3).Infof(\"Orphaned pod %q found, removing\", uid)\r\n\t\tif err := os.RemoveAll(kl.getPodDir(uid)); err != nil {\r\n\t\t\tglog.Infof(\"Failed to remove orphaned pod %q dir; err: %v\", uid, err)\r\n\t\t\terrlist = append(errlist, err)\r\n\t\t}\r\n\t}\r\n\treturn utilerrors.NewAggregate(errlist)\r\n}\r\n```\r\n\r\n>As you can see it gets pods from the disk (by default from `/var/lib/kubelet/pods`) and as they don't have any reference in `etcd` they are marked to be removed. Before removal it is checked if they have any mounted volumes (in `podVolumesExist` method). However, we think (check following listing), that `podVolumesExist` checks only if volumes entries exist in `ectd`.\r\n\r\n>```\r\n$ mount\r\n...\r\ntmpfs on /var/lib/kubelet/pods/bd3f9c64-69d9-11e6-ac66-901b0e5324e4/volumes/kubernetes.io~secret/default-token-i49yf type tmpfs (rw,relatime)\r\ntmpfs on /var/lib/kubelet/pods/0c52522f-69d9-11e6-ac66-901b0e5324e4/volumes/kubernetes.io~secret/default-token-i49yf type tmpfs (rw,relatime)\r\ntmpfs on /var/lib/kubelet/pods/bd39a392-69d9-11e6-ac66-901b0e5324e4/volumes/kubernetes.io~secret/default-token-qzpqq type tmpfs (rw,relatime)\r\ntmpfs on /var/lib/kubelet/pods/bd35515e-69d9-11e6-ac66-901b0e5324e4/volumes/kubernetes.io~secret/default-token-qzpqq type tmpfs (rw,relatime)\r\ntmpfs on /var/lib/kubelet/pods/bd3b8029-69d9-11e6-ac66-901b0e5324e4/volumes/kubernetes.io~secret/default-token-qzpqq type tmpfs (rw,relatime)\r\ntmpfs on /var/lib/kubelet/pods/bd3b88b6-69d9-11e6-ac66-901b0e5324e4/volumes/kubernetes.io~secret/default-token-qzpqq type tmpfs (rw,relatime)\r\ntmpfs on /var/lib/kubelet/pods/bd37f084-69d9-11e6-ac66-901b0e5324e4/volumes/kubernetes.io~secret/default-token-qzpqq type tmpfs (rw,relatime)\r\ntmpfs on /var/lib/kubelet/pods/bd367d32-69d9-11e6-ac66-901b0e5324e4/volumes/kubernetes.io~secret/default-token-qzpqq type tmpfs (rw,relatime)\r\ntmpfs on /var/lib/kubelet/pods/bd396cb4-69d9-11e6-ac66-901b0e5324e4/volumes/kubernetes.io~secret/default-token-qzpqq type tmpfs (rw,relatime)\r\ntmpfs on /var/lib/kubelet/pods/bd3698cb-69d9-11e6-ac66-901b0e5324e4/volumes/kubernetes.io~secret/default-token-qzpqq type tmpfs (rw,relatime)\r\ntmpfs on /var/lib/kubelet/pods/bd38d157-69d9-11e6-ac66-901b0e5324e4/volumes/kubernetes.io~secret/default-token-qzpqq type tmpfs (rw,relatime)\r\ntmpfs on /var/lib/kubelet/pods/bd369312-69d9-11e6-ac66-901b0e5324e4/volumes/kubernetes.io~secret/default-token-qzpqq type tmpfs (rw,relatime)\r\n```\r\n\r\n>It causes error, because `kubelet` wants to remove non-empty directories with mounted volumes.\r\n\r\n>![image](https://cloud.githubusercontent.com/assets/2823399/17965264/f30621b8-6abb-11e6-8fe4-b221127be2b6.png)\r\n\r\n>As a fix for this issue, we think, `etcd` could be synchronized with disk state or update of `cleanupOrphanedPodDirs` function (adding disk state check and unmounting pods before they are removed as you suggested).\r\n\r\n>Should we create issue in the core repository?"}}
{"actor":{"display_login":"maciaszczykm"}}