{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/23742","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/23742/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/23742/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/23742/events","html_url":"https://github.com/kubernetes/kubernetes/issues/23742","id":145233470,"number":23742,"title":"kubelet will never terminate a pod if image registry is unavailable for a pod container","user":{"login":"derekwaynecarr","id":6233452,"avatar_url":"https://avatars.githubusercontent.com/u/6233452?v=3","gravatar_id":"","url":"https://api.github.com/users/derekwaynecarr","html_url":"https://github.com/derekwaynecarr","followers_url":"https://api.github.com/users/derekwaynecarr/followers","following_url":"https://api.github.com/users/derekwaynecarr/following{/other_user}","gists_url":"https://api.github.com/users/derekwaynecarr/gists{/gist_id}","starred_url":"https://api.github.com/users/derekwaynecarr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/derekwaynecarr/subscriptions","organizations_url":"https://api.github.com/users/derekwaynecarr/orgs","repos_url":"https://api.github.com/users/derekwaynecarr/repos","events_url":"https://api.github.com/users/derekwaynecarr/events{/privacy}","received_events_url":"https://api.github.com/users/derekwaynecarr/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/priority/P1","name":"priority/P1","color":"eb6420"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/node","name":"team/node","color":"d2b48c"}],"state":"open","locked":false,"assignee":{"login":"derekwaynecarr","id":6233452,"avatar_url":"https://avatars.githubusercontent.com/u/6233452?v=3","gravatar_id":"","url":"https://api.github.com/users/derekwaynecarr","html_url":"https://github.com/derekwaynecarr","followers_url":"https://api.github.com/users/derekwaynecarr/followers","following_url":"https://api.github.com/users/derekwaynecarr/following{/other_user}","gists_url":"https://api.github.com/users/derekwaynecarr/gists{/gist_id}","starred_url":"https://api.github.com/users/derekwaynecarr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/derekwaynecarr/subscriptions","organizations_url":"https://api.github.com/users/derekwaynecarr/orgs","repos_url":"https://api.github.com/users/derekwaynecarr/repos","events_url":"https://api.github.com/users/derekwaynecarr/events{/privacy}","received_events_url":"https://api.github.com/users/derekwaynecarr/received_events","type":"User","site_admin":false},"milestone":null,"comments":5,"created_at":"2016-04-01T17:10:12Z","updated_at":"2016-04-01T17:33:40Z","closed_at":null,"body":"After an unbelievable amount of time debugging the following this past week:\r\n\r\nhttps://github.com/kubernetes/kubernetes/issues/22045\r\nhttps://github.com/openshift/origin/issues/8176\r\n\r\nI discovered that I could reliably cause the `kubelet` to never delete a pod that was `Terminating` if the pod had never been started correctly because the docker image pull returned a `RegistryUnavailable` error.\r\n\r\nI verified this by modifying the kubelet to always return `RegistryUnavailable` for a particular image name.  I then created a namespace, added a replication controller that referenced the image, and then watched as the pods got stuck in \"Waiting\" state but never actually reported a reason.  I then deleted the namespace, and was able to see that in-fact the `kubelet` never properly handled the `terminated` response.\r\n\r\nI was able to fix this issue by modifying:\r\n\r\nhttps://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/container/serialized_image_puller.go#L124\r\n\r\nto return `ErrImagePull` when the repository was not available.  This change resulted in letting my `kubelet` properly terminate the pod.  I think this is the correct behavior to not special case `RegistryUnavailable` since in many cases the registry may never become available (its most definitely not a temporary situation).\r\n\r\nI suspect this is the potential source of a number of instances of the namespace failed to delete flake because pods remained.\r\n\r\n/cc @kubernetes/sig-node @vishh @dchen1107 @yujuhong @kubernetes/rh-cluster-infra "},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/204481856","html_url":"https://github.com/kubernetes/kubernetes/issues/23742#issuecomment-204481856","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/23742","id":204481856,"user":{"login":"Random-Liu","id":5821883,"avatar_url":"https://avatars.githubusercontent.com/u/5821883?v=3","gravatar_id":"","url":"https://api.github.com/users/Random-Liu","html_url":"https://github.com/Random-Liu","followers_url":"https://api.github.com/users/Random-Liu/followers","following_url":"https://api.github.com/users/Random-Liu/following{/other_user}","gists_url":"https://api.github.com/users/Random-Liu/gists{/gist_id}","starred_url":"https://api.github.com/users/Random-Liu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Random-Liu/subscriptions","organizations_url":"https://api.github.com/users/Random-Liu/orgs","repos_url":"https://api.github.com/users/Random-Liu/repos","events_url":"https://api.github.com/users/Random-Liu/events{/privacy}","received_events_url":"https://api.github.com/users/Random-Liu/received_events","type":"User","site_admin":false},"created_at":"2016-04-01T17:33:39Z","updated_at":"2016-04-01T17:33:39Z","body":"This is related to #21125."}}