{"action":"closed","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/75617","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/75617/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/75617/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/75617/events","html_url":"https://github.com/kubernetes/kubernetes/issues/75617","id":424410497,"node_id":"MDU6SXNzdWU0MjQ0MTA0OTc=","number":75617,"title":"Scheduler: affinity predicates read nodes from informer cache instead of scheduler cache","user":{"login":"misterikkit","id":1796778,"node_id":"MDQ6VXNlcjE3OTY3Nzg=","avatar_url":"https://avatars1.githubusercontent.com/u/1796778?v=4","gravatar_id":"","url":"https://api.github.com/users/misterikkit","html_url":"https://github.com/misterikkit","followers_url":"https://api.github.com/users/misterikkit/followers","following_url":"https://api.github.com/users/misterikkit/following{/other_user}","gists_url":"https://api.github.com/users/misterikkit/gists{/gist_id}","starred_url":"https://api.github.com/users/misterikkit/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/misterikkit/subscriptions","organizations_url":"https://api.github.com/users/misterikkit/orgs","repos_url":"https://api.github.com/users/misterikkit/repos","events_url":"https://api.github.com/users/misterikkit/events{/privacy}","received_events_url":"https://api.github.com/users/misterikkit/received_events","type":"User","site_admin":false},"labels":[{"id":105146071,"node_id":"MDU6TGFiZWwxMDUxNDYwNzE=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/kind/bug","name":"kind/bug","color":"e11d21","default":false},{"id":125550211,"node_id":"MDU6TGFiZWwxMjU1NTAyMTE=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/scheduling","name":"sig/scheduling","color":"d2b48c","default":false}],"state":"closed","locked":false,"assignee":{"login":"draveness","id":6493255,"node_id":"MDQ6VXNlcjY0OTMyNTU=","avatar_url":"https://avatars0.githubusercontent.com/u/6493255?v=4","gravatar_id":"","url":"https://api.github.com/users/draveness","html_url":"https://github.com/draveness","followers_url":"https://api.github.com/users/draveness/followers","following_url":"https://api.github.com/users/draveness/following{/other_user}","gists_url":"https://api.github.com/users/draveness/gists{/gist_id}","starred_url":"https://api.github.com/users/draveness/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/draveness/subscriptions","organizations_url":"https://api.github.com/users/draveness/orgs","repos_url":"https://api.github.com/users/draveness/repos","events_url":"https://api.github.com/users/draveness/events{/privacy}","received_events_url":"https://api.github.com/users/draveness/received_events","type":"User","site_admin":false},"assignees":[{"login":"draveness","id":6493255,"node_id":"MDQ6VXNlcjY0OTMyNTU=","avatar_url":"https://avatars0.githubusercontent.com/u/6493255?v=4","gravatar_id":"","url":"https://api.github.com/users/draveness","html_url":"https://github.com/draveness","followers_url":"https://api.github.com/users/draveness/followers","following_url":"https://api.github.com/users/draveness/following{/other_user}","gists_url":"https://api.github.com/users/draveness/gists{/gist_id}","starred_url":"https://api.github.com/users/draveness/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/draveness/subscriptions","organizations_url":"https://api.github.com/users/draveness/orgs","repos_url":"https://api.github.com/users/draveness/repos","events_url":"https://api.github.com/users/draveness/events{/privacy}","received_events_url":"https://api.github.com/users/draveness/received_events","type":"User","site_admin":false}],"milestone":null,"comments":13,"created_at":"2019-03-22T21:42:20Z","updated_at":"2019-07-01T22:09:34Z","closed_at":"2019-07-01T22:09:34Z","author_association":"CONTRIBUTOR","body":"<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!\r\n\r\nIf the matter is security related, please disclose it privately via https://kubernetes.io/security/\r\n-->\r\n\r\n\r\n**What happened**:\r\nI recently saw an issue in which the scheduler hit this log line many times: \r\nhttps://github.com/kubernetes/kubernetes/blob/6f482974b76db3f1e0f5d24605a9d1d38fad9a2b/pkg/scheduler/algorithm/predicates/predicates.go#L1276\r\n\r\nAfter some digging, it turns out that PodAffinityChecker has a `PodLister` and `NodeInfo`. The `PodLister` points at the scheduler's cache instance, while the `NodeInfo` points at a light wrapper around a `corelisters.NodeLister`.\r\n\r\nhttps://github.com/kubernetes/kubernetes/blob/6f482974b76db3f1e0f5d24605a9d1d38fad9a2b/pkg/scheduler/factory/factory.go#L1329\r\n\r\nIn the scheduler, we use the cache to provide a globally consistent view of the cluster, even if that view is slightly out of date. Data read from the core `NodeInfo` may not have been processed by the scheduler cache yet. Letting data \"skip ahead\" like this can cause stale info to get persisted in the scheduler cache, and other subtle bugs.\r\n\r\nWhile the original problem is not likely caused by this issue, I think it needs to be cleaned up to avoid potential bugs down the line.\r\n\r\n\r\n**What you expected to happen**:\r\nPredicates should only read state from the scheduler's cache in order to maintain consistency.\r\n\r\n**How to reproduce it (as minimally and precisely as possible)**:\r\nThis was observed in a cluster with high rate of pod creation and node creation/deletion.\r\n\r\n**Anything else we need to know?**:\r\n\r\n**Environment**:\r\n- Kubernetes version (use `kubectl version`): 1.12.5\r\n- Cloud provider or hardware configuration: GKE with preemptible nodes\r\n- OS (e.g: `cat /etc/os-release`):\r\n- Kernel (e.g. `uname -a`):\r\n- Install tools:\r\n- Others:\r\n"}}
{"actor":{"display_login":"k8s-ci-robot"}}