{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/38862","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/38862/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/38862/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/38862/events","html_url":"https://github.com/kubernetes/kubernetes/issues/38862","id":196015963,"number":38862,"title":"Kubelet rejects the pod scheduled on it","user":{"login":"hex108","id":765591,"avatar_url":"https://avatars3.githubusercontent.com/u/765591?v=4","gravatar_id":"","url":"https://api.github.com/users/hex108","html_url":"https://github.com/hex108","followers_url":"https://api.github.com/users/hex108/followers","following_url":"https://api.github.com/users/hex108/following{/other_user}","gists_url":"https://api.github.com/users/hex108/gists{/gist_id}","starred_url":"https://api.github.com/users/hex108/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hex108/subscriptions","organizations_url":"https://api.github.com/users/hex108/orgs","repos_url":"https://api.github.com/users/hex108/repos","events_url":"https://api.github.com/users/hex108/events{/privacy}","received_events_url":"https://api.github.com/users/hex108/received_events","type":"User","site_admin":false},"labels":[{"id":116719829,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/area/kubelet","name":"area/kubelet","color":"0052cc","default":false},{"id":778118402,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/lifecycle/rotten","name":"lifecycle/rotten","color":"604460","default":false},{"id":125550211,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/scheduling","name":"sig/scheduling","color":"d2b48c","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2016-12-16T09:31:49Z","updated_at":"2018-03-01T04:57:49Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"We use kubernetes 1.4.0.\r\n\r\nKube-scheduler schedules a pod on a kubelet node. However the kubelet rejects the pod because of OutOfmemory. There is no mirror pod in the cluster and all pods has resource requests, is there any case might cause it?\r\n\r\nKube-scheduler log is:\r\n\r\n>I1214 16:54:26.850177   18367 scheduler.go:96] Attempting to schedule pod: windyliu1/windyliu1-209600820-9d2c1\r\n>I1214 16:54:26.850430   18367 priorities.go:51] Combined requested resources 7247757312 from existing pods exceeds capacity 7125270528 on node node-10-199-15-220\r\nI1214 16:54:26.850813   18367 factory.go:582] Attempting to bind windyliu1-209600820-9d2c1 to node-10-199-15-220\r\n> ...\r\n>I1214 16:54:27.335881   18367 event.go:217] Event(api.ObjectReference{Kind:\"Pod\", Namespace:\"windyliu1\", Name:\"windyliu1-209600820-9d2c1\", UID:\"43c96b33-c1d7-11e6-9e8c-b499baace206\", APIVersion:\"v1\", ResourceVersion:\"1039981\", FieldPath:\"\"}): type: 'Normal' reason: 'Scheduled' Successfully assigned windyliu1-209600820-9d2c1 to node-10-199-15-220\r\n</quote>\r\n\r\nQuestion: we could see that scheduler found there is no enough resource for the pod when making priorities to nodes. All pods has resource requests, so nodes' NonZeroRequests equals normal requests, does it mean that the allocatable resources changes between predicates and priorities?\r\n\r\nKubelet log is:\r\n\r\n> I1214 16:54:46.056416   10497 kubelet.go:2318] SyncLoop (ADD, \"api\"): \"windyliu1-209600820-9d2c1_windyliu1(43c96b33-c1d7-11e6-9e8c-b499baace206)\"\r\n> ...\r\n> I1214 16:54:46.111987   10497 kubelet.go:2223] Predicate failed on Pod: windyliu1-209600820-9d2c1_windyliu1(43c96b33-c1d7-11e6-9e8c-b499baace206), for reason: Node didn't have enough resource: memory, requested: 134217728, used: 7113539584, capacity: 7125270528\r\n> I1214 16:54:46.112442   10497 server.go:608] Event(api.ObjectReference{Kind:\"Pod\", Namespace:\"windyliu1\", Name:\"windyliu1-209600820-9d2c1\", UID:\"43c96b33-c1d7-11e6-9e8c-b499baace206\", APIVersion:\"v1\", ResourceVersion:\"1053804\", FieldPath:\"\"}): type: 'Warning' reason: 'OutOfmemory'\r\n\r\nNote: memory requested(134217728) + used(7113539584) = 7247757312 (The number has been reported by kube-scheduler in above log:  Combined requested resources 7247757312 from existing pods exceeds capacity 7125270528)."},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/369474191","html_url":"https://github.com/kubernetes/kubernetes/issues/38862#issuecomment-369474191","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/38862","id":369474191,"user":{"login":"fejta-bot","id":30488982,"avatar_url":"https://avatars1.githubusercontent.com/u/30488982?v=4","gravatar_id":"","url":"https://api.github.com/users/fejta-bot","html_url":"https://github.com/fejta-bot","followers_url":"https://api.github.com/users/fejta-bot/followers","following_url":"https://api.github.com/users/fejta-bot/following{/other_user}","gists_url":"https://api.github.com/users/fejta-bot/gists{/gist_id}","starred_url":"https://api.github.com/users/fejta-bot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fejta-bot/subscriptions","organizations_url":"https://api.github.com/users/fejta-bot/orgs","repos_url":"https://api.github.com/users/fejta-bot/repos","events_url":"https://api.github.com/users/fejta-bot/events{/privacy}","received_events_url":"https://api.github.com/users/fejta-bot/received_events","type":"User","site_admin":false},"created_at":"2018-03-01T04:57:49Z","updated_at":"2018-03-01T04:57:49Z","author_association":"NONE","body":"Rotten issues close after 30d of inactivity.\nReopen the issue with `/reopen`.\nMark the issue as fresh with `/remove-lifecycle rotten`.\n\nSend feedback to sig-testing, kubernetes/test-infra and/or [fejta](https://github.com/fejta).\n/close"}}
{"actor":{"display_login":"fejta-bot"}}