{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/25056","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/25056/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/25056/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/25056/events","html_url":"https://github.com/kubernetes/kubernetes/issues/25056","id":152667658,"number":25056,"title":"Pod scheduling grinds to a halt on v1.2.3","user":{"login":"apatil","id":52041,"avatar_url":"https://avatars.githubusercontent.com/u/52041?v=3","gravatar_id":"","url":"https://api.github.com/users/apatil","html_url":"https://github.com/apatil","followers_url":"https://api.github.com/users/apatil/followers","following_url":"https://api.github.com/users/apatil/following{/other_user}","gists_url":"https://api.github.com/users/apatil/gists{/gist_id}","starred_url":"https://api.github.com/users/apatil/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apatil/subscriptions","organizations_url":"https://api.github.com/users/apatil/orgs","repos_url":"https://api.github.com/users/apatil/repos","events_url":"https://api.github.com/users/apatil/events{/privacy}","received_events_url":"https://api.github.com/users/apatil/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/kind/support","name":"kind/support","color":"eb6420"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-05-02T23:20:56Z","updated_at":"2016-05-03T00:12:49Z","closed_at":null,"body":"I'm running Kubernetes v1.2.3 using the single-node instructions here: http://kubernetes.io/docs/getting-started-guides/docker/ with Docker v1.10. Everything worked as expected for several hours, but now pods are not being scheduled or removed.\r\n\r\nI've attempted to update a deployment several times over the last hour. Here is `kubectl get pods` for that deployment:\r\n\r\n```\r\n$ kubectl get pods -l role=nfs-server\r\nNAME                          READY     STATUS              RESTARTS   AGE\r\nnfs-server-1605536338-ff5dp   0/1       Terminating         0          26m\r\nnfs-server-2929273945-87ir0   0/1       Terminating         0          10m\r\nnfs-server-3650062886-7regp   0/1       Terminating         0          24m\r\nnfs-server-377129609-3blnt    0/1       Terminating         0          17m\r\nnfs-server-4135136443-iqh32   0/1       ContainerCreating   0          4m\r\nnfs-server-4135136443-ny1op   0/1       Terminating         0          6m\r\n```\r\n\r\nThe ContainerCreating pod's last event from `kubectl describe` is:\r\n\r\n```\r\nEvents:\r\n  FirstSeen     LastSeen        Count   From                    SubobjectPath   Type    Reason           Message\r\n  ---------     --------        -----   ----                    -------------   --------------           -------\r\n  4m            4m              1       {default-scheduler }                    Normal  Scheduled        Successfully assigned nfs-server-4135136443-l3fr4 to 127.0.0.1\r\n```\r\n\r\nWhen I delete a pod, the scheduler does seem to start the new pod almost immediately, and the kubelet logs:\r\n\r\n```\r\nI0502 23:13:28.627867    1701 kubelet.go:2388] SyncLoop (ADD, \"api\"): \"nfs-server-4135136443-woha5_default(7ac455d2-10bb-11e6-b93b-06b60bd2d431)\"  \r\n```\r\n\r\nThe kubelet then begins logging a readiness check failure every 10s:\r\n\r\n```\r\nI0502 23:14:27.160982    1701 prober.go:104] Readiness probe for \"nfs-server-1605536338-ff5dp_default(8f573313-10b7-11e6-b93b-06b60bd2d431):nfs-server\" failed (failure): dial tcp 172.19.0.2:111: i/o timeout\r\n```\r\n\r\nbut I can't see any evidence that it actually tries to run the container. Docker doesn't emit any logs. The Docker daemon is responsive, and the image is definitely present. When I run the image using `docker run` it works fine.\r\n\r\nAnything I can check?\r\n\r\nThanks!"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/216401432","html_url":"https://github.com/kubernetes/kubernetes/issues/25056#issuecomment-216401432","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/25056","id":216401432,"user":{"login":"yujuhong","id":10524058,"avatar_url":"https://avatars.githubusercontent.com/u/10524058?v=3","gravatar_id":"","url":"https://api.github.com/users/yujuhong","html_url":"https://github.com/yujuhong","followers_url":"https://api.github.com/users/yujuhong/followers","following_url":"https://api.github.com/users/yujuhong/following{/other_user}","gists_url":"https://api.github.com/users/yujuhong/gists{/gist_id}","starred_url":"https://api.github.com/users/yujuhong/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yujuhong/subscriptions","organizations_url":"https://api.github.com/users/yujuhong/orgs","repos_url":"https://api.github.com/users/yujuhong/repos","events_url":"https://api.github.com/users/yujuhong/events{/privacy}","received_events_url":"https://api.github.com/users/yujuhong/received_events","type":"User","site_admin":false},"created_at":"2016-05-03T00:12:49Z","updated_at":"2016-05-03T00:12:49Z","body":"@apatil, is there any message in the kubelet log related to image pulling? Without further information, it seems like kubelet could be stuck at pulling the images. It'd be helpful to see if there are related information."}}