{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/37447","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/37447/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/37447/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/37447/events","html_url":"https://github.com/kubernetes/kubernetes/issues/37447","id":191519197,"number":37447,"title":"Upon restart, kubelet tries to start all pods including terminated ones","user":{"login":"fgrzadkowski","id":10820726,"avatar_url":"https://avatars.githubusercontent.com/u/10820726?v=3","gravatar_id":"","url":"https://api.github.com/users/fgrzadkowski","html_url":"https://github.com/fgrzadkowski","followers_url":"https://api.github.com/users/fgrzadkowski/followers","following_url":"https://api.github.com/users/fgrzadkowski/following{/other_user}","gists_url":"https://api.github.com/users/fgrzadkowski/gists{/gist_id}","starred_url":"https://api.github.com/users/fgrzadkowski/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fgrzadkowski/subscriptions","organizations_url":"https://api.github.com/users/fgrzadkowski/orgs","repos_url":"https://api.github.com/users/fgrzadkowski/repos","events_url":"https://api.github.com/users/fgrzadkowski/events{/privacy}","received_events_url":"https://api.github.com/users/fgrzadkowski/received_events","type":"User","site_admin":false},"labels":[{"id":116719829,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/component/kubelet","name":"component/kubelet","color":"0052cc","default":false},{"id":114528223,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/priority/P1","name":"priority/P1","color":"eb6420","default":false},{"id":435667038,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/release-blocker","name":"release-blocker","color":"d93f0b","default":false},{"id":173493665,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/node","name":"team/node","color":"d2b48c","default":false}],"state":"open","locked":false,"assignee":{"login":"dashpole","id":3262098,"avatar_url":"https://avatars.githubusercontent.com/u/3262098?v=3","gravatar_id":"","url":"https://api.github.com/users/dashpole","html_url":"https://github.com/dashpole","followers_url":"https://api.github.com/users/dashpole/followers","following_url":"https://api.github.com/users/dashpole/following{/other_user}","gists_url":"https://api.github.com/users/dashpole/gists{/gist_id}","starred_url":"https://api.github.com/users/dashpole/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dashpole/subscriptions","organizations_url":"https://api.github.com/users/dashpole/orgs","repos_url":"https://api.github.com/users/dashpole/repos","events_url":"https://api.github.com/users/dashpole/events{/privacy}","received_events_url":"https://api.github.com/users/dashpole/received_events","type":"User","site_admin":false},"assignees":[{"login":"dashpole","id":3262098,"avatar_url":"https://avatars.githubusercontent.com/u/3262098?v=3","gravatar_id":"","url":"https://api.github.com/users/dashpole","html_url":"https://github.com/dashpole","followers_url":"https://api.github.com/users/dashpole/followers","following_url":"https://api.github.com/users/dashpole/following{/other_user}","gists_url":"https://api.github.com/users/dashpole/gists{/gist_id}","starred_url":"https://api.github.com/users/dashpole/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dashpole/subscriptions","organizations_url":"https://api.github.com/users/dashpole/orgs","repos_url":"https://api.github.com/users/dashpole/repos","events_url":"https://api.github.com/users/dashpole/events{/privacy}","received_events_url":"https://api.github.com/users/dashpole/received_events","type":"User","site_admin":false},{"login":"dchen1107","id":7740897,"avatar_url":"https://avatars.githubusercontent.com/u/7740897?v=3","gravatar_id":"","url":"https://api.github.com/users/dchen1107","html_url":"https://github.com/dchen1107","followers_url":"https://api.github.com/users/dchen1107/followers","following_url":"https://api.github.com/users/dchen1107/following{/other_user}","gists_url":"https://api.github.com/users/dchen1107/gists{/gist_id}","starred_url":"https://api.github.com/users/dchen1107/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dchen1107/subscriptions","organizations_url":"https://api.github.com/users/dchen1107/orgs","repos_url":"https://api.github.com/users/dchen1107/repos","events_url":"https://api.github.com/users/dchen1107/events{/privacy}","received_events_url":"https://api.github.com/users/dchen1107/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/23","html_url":"https://github.com/kubernetes/kubernetes/milestones/v1.5","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/23/labels","id":1945978,"number":23,"title":"v1.5","description":"","creator":{"login":"lavalamp","id":647318,"avatar_url":"https://avatars.githubusercontent.com/u/647318?v=3","gravatar_id":"","url":"https://api.github.com/users/lavalamp","html_url":"https://github.com/lavalamp","followers_url":"https://api.github.com/users/lavalamp/followers","following_url":"https://api.github.com/users/lavalamp/following{/other_user}","gists_url":"https://api.github.com/users/lavalamp/gists{/gist_id}","starred_url":"https://api.github.com/users/lavalamp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lavalamp/subscriptions","organizations_url":"https://api.github.com/users/lavalamp/orgs","repos_url":"https://api.github.com/users/lavalamp/repos","events_url":"https://api.github.com/users/lavalamp/events{/privacy}","received_events_url":"https://api.github.com/users/lavalamp/received_events","type":"User","site_admin":false},"open_issues":130,"closed_issues":894,"state":"open","created_at":"2016-08-16T21:47:28Z","updated_at":"2016-11-30T20:02:55Z","due_on":"2016-12-09T00:00:00Z","closed_at":null},"comments":20,"created_at":"2016-11-24T13:23:17Z","updated_at":"2016-11-30T20:12:34Z","closed_at":null,"body":"<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.-->\r\n\r\n**Is this a request for help?** (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):\r\n\r\n\r\n**What keywords did you search in Kubernetes issues before filing this one?** (If you have found any duplicates, you should instead reply there.):\r\n\r\n---\r\n\r\n**Is this a BUG REPORT or FEATURE REQUEST?** (choose one):\r\n\r\n<!--\r\nIf this is a BUG REPORT, please:\r\n  - Fill in as much of the template below as you can.  If you leave out\r\n    information, we can't help you as well.\r\n\r\nIf this is a FEATURE REQUEST, please:\r\n  - Describe *in detail* the feature/behavior/change you'd like to see.\r\n\r\nIn both cases, be ready for followup questions, and please respond in a timely\r\nmanner.  If we can't reproduce a bug or think a feature already exists, we\r\nmight close your issue.  If we're wrong, PLEASE feel free to reopen it and\r\nexplain why.\r\n-->\r\n\r\n**Kubernetes version** (use `kubectl version`):\r\n\r\n```\r\n$ kubectl version\r\nClient Version: version.Info{Major:\"1\", Minor:\"3\", GitVersion:\"v1.3.6\", GitCommit:\"ae4550cc9c89a593bcda6678df201db1b208133b\", GitTreeState:\"clean\", BuildDate:\"2016-08-26T18:13:23Z\", GoVersion:\"go1.6.2\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\nServer Version: version.Info{Major:\"1\", Minor:\"4\", GitVersion:\"v1.4.6\", GitCommit:\"e569a27d02001e343cb68086bc06d47804f62af6\", GitTreeState:\"clean\", BuildDate:\"2016-11-12T05:16:27Z\", GoVersion:\"go1.6.3\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\n```\r\n\r\n**Environment**:\r\n- **Cloud provider or hardware configuration**: GKE\r\n- **OS** (e.g. from /etc/os-release): default\r\n- **Kernel** (e.g. `uname -a`):\r\n- **Install tools**:\r\n- **Others**:\r\n\r\n\r\n**What happened**:\r\nI was trying to reproduce https://github.com/kubernetes/kubernetes/issues/34772 using [these instructions](https://github.com/kubernetes/kubernetes/issues/34772#issuecomment-260594667). Due to bug in scheduler we were assigning too many pods per node. In most cases kubelet was correctly discarding them by marking them as `OutOfcpu`. However two strange things happened:\r\n* upon kubelet restart, it tried to re-add all such `OutOfcpu` pods, run predicate check for them etc.\r\n* for two pods that were incorrectly assigned (due to scheduler bug) during kubelet restart kubelet completely ignored them - they are still pending, with node assigned for 12h.\r\n\r\nI have a cluster that I can add someone to investigate.\r\n\r\n**What you expected to happen**:\r\n\r\n* `OutOfcpu` pods should not be retried after restart\r\n* new pods, assigned during kubelet restart, should be discarded with `OutOfcpu` error.\r\n\r\n**How to reproduce it** (as minimally and precisely as possible):\r\n\r\nRun [these instructions](https://github.com/kubernetes/kubernetes/issues/34772#issuecomment-260594667).\r\n\r\n**Anything else do we need to know**:\r\n\r\nMarking as `release-blocker` until it's triaged by node team.\r\n"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/263981868","html_url":"https://github.com/kubernetes/kubernetes/issues/37447#issuecomment-263981868","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/37447","id":263981868,"user":{"login":"dashpole","id":3262098,"avatar_url":"https://avatars.githubusercontent.com/u/3262098?v=3","gravatar_id":"","url":"https://api.github.com/users/dashpole","html_url":"https://github.com/dashpole","followers_url":"https://api.github.com/users/dashpole/followers","following_url":"https://api.github.com/users/dashpole/following{/other_user}","gists_url":"https://api.github.com/users/dashpole/gists{/gist_id}","starred_url":"https://api.github.com/users/dashpole/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dashpole/subscriptions","organizations_url":"https://api.github.com/users/dashpole/orgs","repos_url":"https://api.github.com/users/dashpole/repos","events_url":"https://api.github.com/users/dashpole/events{/privacy}","received_events_url":"https://api.github.com/users/dashpole/received_events","type":"User","site_admin":false},"created_at":"2016-11-30T20:12:34Z","updated_at":"2016-11-30T20:12:34Z","body":"@yujuhong 's fix appears to prevent pods from being permanently pending.  While all pods (including terminated pods) are re-added to the kubelet's internal pod manager after restarting, it does not appear that it \"processes\" rejected pods a second time."}}
{"actor":{"display_login":"dashpole"}}