{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/autoscaler/issues/257","repository_url":"https://api.github.com/repos/kubernetes/autoscaler","labels_url":"https://api.github.com/repos/kubernetes/autoscaler/issues/257/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/autoscaler/issues/257/comments","events_url":"https://api.github.com/repos/kubernetes/autoscaler/issues/257/events","html_url":"https://github.com/kubernetes/autoscaler/issues/257","id":252590542,"number":257,"title":"Figure out and implement custom handling for MatchInterPodAffinity predicate","user":{"login":"MaciekPytel","id":7901030,"avatar_url":"https://avatars2.githubusercontent.com/u/7901030?v=4","gravatar_id":"","url":"https://api.github.com/users/MaciekPytel","html_url":"https://github.com/MaciekPytel","followers_url":"https://api.github.com/users/MaciekPytel/followers","following_url":"https://api.github.com/users/MaciekPytel/following{/other_user}","gists_url":"https://api.github.com/users/MaciekPytel/gists{/gist_id}","starred_url":"https://api.github.com/users/MaciekPytel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MaciekPytel/subscriptions","organizations_url":"https://api.github.com/users/MaciekPytel/orgs","repos_url":"https://api.github.com/users/MaciekPytel/repos","events_url":"https://api.github.com/users/MaciekPytel/events{/privacy}","received_events_url":"https://api.github.com/users/MaciekPytel/received_events","type":"User","site_admin":false},"labels":[{"id":581899376,"url":"https://api.github.com/repos/kubernetes/autoscaler/labels/bug","name":"bug","color":"ee0701","default":true},{"id":587167715,"url":"https://api.github.com/repos/kubernetes/autoscaler/labels/cluster-autoscaler","name":"cluster-autoscaler","color":"d4c5f9","default":false},{"id":581899378,"url":"https://api.github.com/repos/kubernetes/autoscaler/labels/enhancement","name":"enhancement","color":"84b6eb","default":true},{"id":778120204,"url":"https://api.github.com/repos/kubernetes/autoscaler/labels/lifecycle/frozen","name":"lifecycle/frozen","color":"d3e2f0","default":false}],"state":"open","locked":false,"assignee":{"login":"MaciekPytel","id":7901030,"avatar_url":"https://avatars2.githubusercontent.com/u/7901030?v=4","gravatar_id":"","url":"https://api.github.com/users/MaciekPytel","html_url":"https://github.com/MaciekPytel","followers_url":"https://api.github.com/users/MaciekPytel/followers","following_url":"https://api.github.com/users/MaciekPytel/following{/other_user}","gists_url":"https://api.github.com/users/MaciekPytel/gists{/gist_id}","starred_url":"https://api.github.com/users/MaciekPytel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MaciekPytel/subscriptions","organizations_url":"https://api.github.com/users/MaciekPytel/orgs","repos_url":"https://api.github.com/users/MaciekPytel/repos","events_url":"https://api.github.com/users/MaciekPytel/events{/privacy}","received_events_url":"https://api.github.com/users/MaciekPytel/received_events","type":"User","site_admin":false},"assignees":[{"login":"MaciekPytel","id":7901030,"avatar_url":"https://avatars2.githubusercontent.com/u/7901030?v=4","gravatar_id":"","url":"https://api.github.com/users/MaciekPytel","html_url":"https://github.com/MaciekPytel","followers_url":"https://api.github.com/users/MaciekPytel/followers","following_url":"https://api.github.com/users/MaciekPytel/following{/other_user}","gists_url":"https://api.github.com/users/MaciekPytel/gists{/gist_id}","starred_url":"https://api.github.com/users/MaciekPytel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MaciekPytel/subscriptions","organizations_url":"https://api.github.com/users/MaciekPytel/orgs","repos_url":"https://api.github.com/users/MaciekPytel/repos","events_url":"https://api.github.com/users/MaciekPytel/events{/privacy}","received_events_url":"https://api.github.com/users/MaciekPytel/received_events","type":"User","site_admin":false}],"milestone":null,"comments":13,"created_at":"2017-08-24T12:48:39Z","updated_at":"2018-02-09T14:31:25Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"As part of working on CA performance we run a large scale-up test with some additional logging including call count and total duration spent in each predicate. The results are as follows: \r\n```\r\nE0824 09:27:54.815425       8 predicates.go:192] Predicate statistics for MaxAzureDiskVolumeCount: called 59985 times, total time 25.328538ms, mean duration 422ns\r\nE0824 09:27:54.815489       8 predicates.go:192] Predicate statistics for MatchInterPodAffinity: called 59985 times, total time 1m48.73252767s, mean duration 1.812661ms\r\nE0824 09:27:54.815497       8 predicates.go:192] Predicate statistics for CheckNodeCondition: called 59985 times, total time 49.05121ms, mean duration 817ns\r\nE0824 09:27:54.815502       8 predicates.go:192] Predicate statistics for MaxEBSVolumeCount: called 59985 times, total time 24.906838ms, mean duration 415ns\r\nE0824 09:27:54.815508       8 predicates.go:192] Predicate statistics for GeneralPredicates: called 59985 times, total time 114.434325ms, mean duration 1.907µs\r\nE0824 09:27:54.815534       8 predicates.go:192] Predicate statistics for NoDiskConflict: called 59985 times, total time 26.067526ms, mean duration 434ns\r\nE0824 09:27:54.815553       8 predicates.go:192] Predicate statistics for NoVolumeNodeConflict: called 59985 times, total time 38.554035ms, mean duration 642ns\r\nE0824 09:27:54.815559       8 predicates.go:192] Predicate statistics for CheckNodeDiskPressure: called 59985 times, total time 19.062642ms, mean duration 317ns\r\nE0824 09:27:54.815564       8 predicates.go:192] Predicate statistics for PodToleratesNodeTaints: called 59985 times, total time 22.448605ms, mean duration 374ns\r\nE0824 09:27:54.815568       8 predicates.go:192] Predicate statistics for MaxGCEPDVolumeCount: called 59985 times, total time 61.944698ms, mean duration 1.032µs\r\nE0824 09:27:54.815572       8 predicates.go:192] Predicate statistics for NoVolumeZoneConflict: called 59985 times, total time 64.231254ms, mean duration 1.07µs \r\nE0824 09:27:54.815578       8 predicates.go:192] Predicate statistics for PodFitsResources: called 357952 times, total time 514.838808ms, mean duration 1.438µs\r\nE0824 09:27:54.815584       8 predicates.go:192] Predicate statistics for ready: called 59985 times, total time 50.931691ms, mean duration 849ns\r\nE0824 09:27:54.815588       8 predicates.go:192] Predicate statistics for CheckNodeMemoryPressure: called 59985 times, total time 285.070472ms, mean duration 4.752µs\r\n```\r\n\r\nIt turns out that MatchInterPodAffinity predicate is **3 orders of magnitude** slower compared to other predicates. This is likely because contrary to scheduler we don't do any precomputation for it and we don't maintain predicateMeta object. \r\n\r\nAfter a quick glance at predicate code it makes sense - it needs to iterate over all existing pods to check if any of them has pod antiaffinity on the pod we're running predicates for. This brings up another problem - how does it get all pods and nodes? We only provide NodeInfo for a single node, the rest comes out of informer. However, that means it reflects the real state of the cluster, not our simulated state. If we've already placed a pod with zone-level antiaffinity on a simulated node it won't prevent adding pods to other simulated nodes in the same zone.\r\n\r\nBottom line is that using zone-level antiaffinity can cause CA to \"overshoot\" creating some nodes for pods that won't be able to schedule on them anyway. Fortunately, this is a pretty unlikely edge case and we will scale-down the unnecessary nodes without any problem."},"comment":{"url":"https://api.github.com/repos/kubernetes/autoscaler/issues/comments/364449232","html_url":"https://github.com/kubernetes/autoscaler/issues/257#issuecomment-364449232","issue_url":"https://api.github.com/repos/kubernetes/autoscaler/issues/257","id":364449232,"user":{"login":"MaciekPytel","id":7901030,"avatar_url":"https://avatars2.githubusercontent.com/u/7901030?v=4","gravatar_id":"","url":"https://api.github.com/users/MaciekPytel","html_url":"https://github.com/MaciekPytel","followers_url":"https://api.github.com/users/MaciekPytel/followers","following_url":"https://api.github.com/users/MaciekPytel/following{/other_user}","gists_url":"https://api.github.com/users/MaciekPytel/gists{/gist_id}","starred_url":"https://api.github.com/users/MaciekPytel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MaciekPytel/subscriptions","organizations_url":"https://api.github.com/users/MaciekPytel/orgs","repos_url":"https://api.github.com/users/MaciekPytel/repos","events_url":"https://api.github.com/users/MaciekPytel/events{/privacy}","received_events_url":"https://api.github.com/users/MaciekPytel/received_events","type":"User","site_admin":false},"created_at":"2018-02-09T14:31:25Z","updated_at":"2018-02-09T14:31:25Z","author_association":"CONTRIBUTOR","body":"@aleksandra-malinowska pointed out to me that a deployment with self-pod-antiaffinity causes nodes to be added one by one. This is because the problem with affinity predicate taking data from informers mentioned in the description of this issue is still not fixed.\r\n\r\nOther optimizations were easier to implement and got us to where we met our performance target and we actually never started using predicateMeta when binpacking. Note that this is trickier than in case of scale-down, because:\r\n1. The state of cluster is constantly changing and we need to update predicateMeta in a way that doesn't cripple our performance and introduce bugs (using stale predicateMeta would break binpacking, likely in subtle and hard to debug ways).\r\n2. Regardless of how smart we are about calculating predicateMeta it will likely still hurt performance if there are no pods using hard pod affinity / antiaffinity. We should make sure this doesn't happen."}}
{"actor":{"display_login":"MaciekPytel"}}