{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/24819","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/24819/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/24819/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/24819/events","html_url":"https://github.com/kubernetes/kubernetes/issues/24819","id":151233611,"number":24819,"title":"Pods can get stuck Terminating in certain situations if running a container fails","user":{"login":"ncdc","id":354,"avatar_url":"https://avatars.githubusercontent.com/u/354?v=3","gravatar_id":"","url":"https://api.github.com/users/ncdc","html_url":"https://github.com/ncdc","followers_url":"https://api.github.com/users/ncdc/followers","following_url":"https://api.github.com/users/ncdc/following{/other_user}","gists_url":"https://api.github.com/users/ncdc/gists{/gist_id}","starred_url":"https://api.github.com/users/ncdc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ncdc/subscriptions","organizations_url":"https://api.github.com/users/ncdc/orgs","repos_url":"https://api.github.com/users/ncdc/repos","events_url":"https://api.github.com/users/ncdc/events{/privacy}","received_events_url":"https://api.github.com/users/ncdc/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/node","name":"team/node","color":"d2b48c"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":11,"created_at":"2016-04-26T20:39:53Z","updated_at":"2016-04-29T19:27:06Z","closed_at":null,"body":"In one of our clusters, we are not allowing containers to run images that use `VOLUME` instructions without a corresponding Kubernetes volume and volumeMount to back it. We've done this by making `/var/lib/docker/volumes` immutable (`chattr +i`). An attempt to run a pod like this results in the container creation falling, and there's no way for the pod to run successfully.\r\n\r\nWhat we're seeing is that sometimes, attempts to delete the pod result in the pod getting stuck Terminating unless you delete it with a grace period of 0. This appears to be because of the following:\r\n\r\n1. syncPod processes new pod and tries to create and run the container whose image has a `VOLUME`\r\n1. while this is happening, the PLEG relisting happens\r\n  1. list all containers\r\n  1. the list includes the \"in-flight\" container (that is about to fail creation)\r\n  1. in-flight container is inspected\r\n  1. inspection fails with something like this: `API error (500): Unknown device 27038b8e8b60506e1744c7ab15e06a3e24855fa925678e75d72b6e1fd3bd4a31`\r\n  1. PLEG writes pod status to pod status cache, which includes the error above\r\n  1. at the end of relisting, PLEG updates the global timestamp for the pod status cache\r\n1. syncPod finishes in error and requeues this pod\r\n1. 1 or more PLEG relistings happen, updating the global timestamp\r\n1. another iteration of the pod worker retrieves the pod status from the cache\r\n  1. the cache's global timestamp is newer than the last sync time for the pod worker, so the cache returns the cached status, which includes the \"Unknown device\" error\r\n  1. because the error is non-nil, the worker exits its loop early, not calling syncPod (https://github.com/kubernetes/kubernetes/blob/392fc6668f5536af71950f7df56fe3f89256a66b/pkg/kubelet/pod_workers.go#L125-L127)\r\n1. because syncPod isn't called, the pod isn't deleted\r\n1. repeat infinitely\r\n\r\n@kubernetes/sig-node @kubernetes/rh-cluster-infra @ironcladlou @smarterclayton "},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/215855133","html_url":"https://github.com/kubernetes/kubernetes/issues/24819#issuecomment-215855133","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/24819","id":215855133,"user":{"login":"ncdc","id":354,"avatar_url":"https://avatars.githubusercontent.com/u/354?v=3","gravatar_id":"","url":"https://api.github.com/users/ncdc","html_url":"https://github.com/ncdc","followers_url":"https://api.github.com/users/ncdc/followers","following_url":"https://api.github.com/users/ncdc/following{/other_user}","gists_url":"https://api.github.com/users/ncdc/gists{/gist_id}","starred_url":"https://api.github.com/users/ncdc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ncdc/subscriptions","organizations_url":"https://api.github.com/users/ncdc/orgs","repos_url":"https://api.github.com/users/ncdc/repos","events_url":"https://api.github.com/users/ncdc/events{/privacy}","received_events_url":"https://api.github.com/users/ncdc/received_events","type":"User","site_admin":false},"created_at":"2016-04-29T19:27:06Z","updated_at":"2016-04-29T19:27:06Z","body":"> I think it's reasonable to assume taht the consumers will retry if the cache is not ready, like what the pod workers do today, as long as this doesn't happen often.\r\n\r\nI'm actually seeing this happen frequently (the presence of the \"Unknown device\" errors during PLEG inspection), but it's not a problem because it's happening during the creation of a container, and a subsequent PLEG relist ends up with a successful inspection.\r\n\r\nI wrote a sample go program to create a container in 1 goroutine and then inspect it as often and as fast as possible until it gets no error. I was able to have the inspection fail 45-50 times with \"Unknown device\" prior to it succeeding. While this isn't really a valid test (hammering the Docker daemon to inspect 1 container), it does highlight that this can certainly happen."}}