{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/31272","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/31272/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/31272/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/31272/events","html_url":"https://github.com/kubernetes/kubernetes/issues/31272","id":172758875,"number":31272,"title":"Hung volumes can wedge the kubelet","user":{"login":"ncdc","id":354,"avatar_url":"https://avatars.githubusercontent.com/u/354?v=3","gravatar_id":"","url":"https://api.github.com/users/ncdc","html_url":"https://github.com/ncdc","followers_url":"https://api.github.com/users/ncdc/followers","following_url":"https://api.github.com/users/ncdc/following{/other_user}","gists_url":"https://api.github.com/users/ncdc/gists{/gist_id}","starred_url":"https://api.github.com/users/ncdc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ncdc/subscriptions","organizations_url":"https://api.github.com/users/ncdc/orgs","repos_url":"https://api.github.com/users/ncdc/repos","events_url":"https://api.github.com/users/ncdc/events{/privacy}","received_events_url":"https://api.github.com/users/ncdc/received_events","type":"User","site_admin":false},"labels":[{"id":169428334,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/area/storage","name":"area/storage","color":"006b75","default":false},{"id":116719829,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/component/kubelet","name":"component/kubelet","color":"0052cc","default":false},{"id":105146071,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/kind/bug","name":"kind/bug","color":"e11d21","default":false},{"id":433683967,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/non-release-blocker","name":"non-release-blocker","color":"0e8a16","default":false},{"id":114528223,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/priority/P1","name":"priority/P1","color":"eb6420","default":false},{"id":173493827,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/cluster","name":"team/cluster","color":"d2b48c","default":false}],"state":"open","locked":false,"assignee":{"login":"pmorie","id":366488,"avatar_url":"https://avatars.githubusercontent.com/u/366488?v=3","gravatar_id":"","url":"https://api.github.com/users/pmorie","html_url":"https://github.com/pmorie","followers_url":"https://api.github.com/users/pmorie/followers","following_url":"https://api.github.com/users/pmorie/following{/other_user}","gists_url":"https://api.github.com/users/pmorie/gists{/gist_id}","starred_url":"https://api.github.com/users/pmorie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pmorie/subscriptions","organizations_url":"https://api.github.com/users/pmorie/orgs","repos_url":"https://api.github.com/users/pmorie/repos","events_url":"https://api.github.com/users/pmorie/events{/privacy}","received_events_url":"https://api.github.com/users/pmorie/received_events","type":"User","site_admin":false},"assignees":[{"login":"pmorie","id":366488,"avatar_url":"https://avatars.githubusercontent.com/u/366488?v=3","gravatar_id":"","url":"https://api.github.com/users/pmorie","html_url":"https://github.com/pmorie","followers_url":"https://api.github.com/users/pmorie/followers","following_url":"https://api.github.com/users/pmorie/following{/other_user}","gists_url":"https://api.github.com/users/pmorie/gists{/gist_id}","starred_url":"https://api.github.com/users/pmorie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pmorie/subscriptions","organizations_url":"https://api.github.com/users/pmorie/orgs","repos_url":"https://api.github.com/users/pmorie/repos","events_url":"https://api.github.com/users/pmorie/events{/privacy}","received_events_url":"https://api.github.com/users/pmorie/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/23","html_url":"https://github.com/kubernetes/kubernetes/milestones/v1.5","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/23/labels","id":1945978,"number":23,"title":"v1.5","description":"","creator":{"login":"lavalamp","id":647318,"avatar_url":"https://avatars.githubusercontent.com/u/647318?v=3","gravatar_id":"","url":"https://api.github.com/users/lavalamp","html_url":"https://github.com/lavalamp","followers_url":"https://api.github.com/users/lavalamp/followers","following_url":"https://api.github.com/users/lavalamp/following{/other_user}","gists_url":"https://api.github.com/users/lavalamp/gists{/gist_id}","starred_url":"https://api.github.com/users/lavalamp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lavalamp/subscriptions","organizations_url":"https://api.github.com/users/lavalamp/orgs","repos_url":"https://api.github.com/users/lavalamp/repos","events_url":"https://api.github.com/users/lavalamp/events{/privacy}","received_events_url":"https://api.github.com/users/lavalamp/received_events","type":"User","site_admin":false},"open_issues":142,"closed_issues":831,"state":"open","created_at":"2016-08-16T21:47:28Z","updated_at":"2016-11-23T18:35:45Z","due_on":"2016-12-09T00:00:00Z","closed_at":null},"comments":76,"created_at":"2016-08-23T17:28:53Z","updated_at":"2016-11-23T18:46:19Z","closed_at":null,"body":"If you have pods that use something like NFS storage, if the system is unable to read the mounted directory, or unmount it, it is possible to completely wedge the kubelet such that it can't successfully run any new pods that use volumes (which is basically all, if they use secret tokens) until either the storage issue is resolved, or you restart the kubelet.\n\nTo reproduce:\n1. Create a pod that uses an NFS volume\n2. Stop the NFS server\n3. Try to delete the pod\n4. Wait a couple of minutes - you'll see the pod stuck Terminating\n5. Try to create a new pod (`kubectl run --rm --attach --restart Never --image busybox bbox date`)\n\nThe busybox pod will be stuck ContainerCreating with events such as these:\n\n```\nEvents:\n  FirstSeen     LastSeen        Count   From                    SubobjectPath   Type            Reason          Message\n  ---------     --------        -----   ----                    -------------   --------        ------          -------\n  8m            8m              1       {default-scheduler }                    Normal          Scheduled       Successfully assigned bbox to 127.0.0.1\n  6m            6s              4       {kubelet 127.0.0.1}                     Warning         FailedMount     Unable to mount volumes for pod \"bbox_default(12a20cdb-694f-11e6-baa4-001c42e13e5d)\": timeout expired waiting for volumes to attach/mount for pod \"bbox\"/\"default\". list of unattached/unmounted volumes=[default-token-joiwi]\n  6m            6s              4       {kubelet 127.0.0.1}                     Warning         FailedSync      Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \"bbox\"/\"default\". list of unattached/unmounted volumes=[default-token-joiwi]\n```\n\nIn this [stack trace](https://gist.github.com/ncdc/d6411b82cd6169991dc3b98c203f59e1#file-stack-deleting-pod-txt-L53-L67) I gathered after I deleted the pod, it shows that the volume reconciler is still trying to get the volumes for the pod I just deleted. You'll also see a goroutine trying to stop the Docker container, but it is stuck.\n\nIn this [stack trace](https://gist.github.com/ncdc/d6411b82cd6169991dc3b98c203f59e1#file-stack-run-pod-txt-L32-L41) I gathered after I tried to create the bbox pod, it shows that the new pod (bbox) is waiting for its volumes to attach/mount (in this case, secrets).\n\nWe've seen this in 1.2.x and I just reproduced it in master (commit f297ea966e0916c18dc008af566975bd26dfb29c).\n\ncc @kubernetes/sig-storage @kubernetes/sig-node @kubernetes/rh-cluster-infra @pmorie @derekwaynecarr @timothysc @saad-ali \n"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/262599512","html_url":"https://github.com/kubernetes/kubernetes/issues/31272#issuecomment-262599512","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/31272","id":262599512,"user":{"login":"eparis","id":8093535,"avatar_url":"https://avatars.githubusercontent.com/u/8093535?v=3","gravatar_id":"","url":"https://api.github.com/users/eparis","html_url":"https://github.com/eparis","followers_url":"https://api.github.com/users/eparis/followers","following_url":"https://api.github.com/users/eparis/following{/other_user}","gists_url":"https://api.github.com/users/eparis/gists{/gist_id}","starred_url":"https://api.github.com/users/eparis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eparis/subscriptions","organizations_url":"https://api.github.com/users/eparis/orgs","repos_url":"https://api.github.com/users/eparis/repos","events_url":"https://api.github.com/users/eparis/events{/privacy}","received_events_url":"https://api.github.com/users/eparis/received_events","type":"User","site_admin":false},"created_at":"2016-11-23T18:46:19Z","updated_at":"2016-11-23T18:46:19Z","body":"Checking the mount table is ok, but on systems with large number of containers it might be HUGE. 2 stat calls are quick. Parsing a 1,000,000+ line file isn't. Paul talked to me about having a go routine which parsed the mount table every X seconds and then provided these answers as a service of some sort. That might be feasible but we have to understand that the answers from this service would be out of date by up to X seconds..."}}
{"actor":{"display_login":"eparis"}}