{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/21672","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/21672/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/21672/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/21672/events","html_url":"https://github.com/kubernetes/kubernetes/issues/21672","id":135398820,"number":21672,"title":"ReplicationControllerManager is too slow to support 1000-node clusters","user":{"login":"wojtek-t","id":10743879,"avatar_url":"https://avatars.githubusercontent.com/u/10743879?v=3","gravatar_id":"","url":"https://api.github.com/users/wojtek-t","html_url":"https://github.com/wojtek-t","followers_url":"https://api.github.com/users/wojtek-t/followers","following_url":"https://api.github.com/users/wojtek-t/following{/other_user}","gists_url":"https://api.github.com/users/wojtek-t/gists{/gist_id}","starred_url":"https://api.github.com/users/wojtek-t/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wojtek-t/subscriptions","organizations_url":"https://api.github.com/users/wojtek-t/orgs","repos_url":"https://api.github.com/users/wojtek-t/repos","events_url":"https://api.github.com/users/wojtek-t/events{/privacy}","received_events_url":"https://api.github.com/users/wojtek-t/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/area/performance","name":"area/performance","color":"d4c5f9"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/component/controller-manager","name":"component/controller-manager","color":"0052cc"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/priority/P1","name":"priority/P1","color":"eb6420"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/control-plane","name":"team/control-plane","color":"d2b48c"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/ux","name":"team/ux","color":"d2b48c"}],"state":"open","locked":false,"assignee":{"login":"wojtek-t","id":10743879,"avatar_url":"https://avatars.githubusercontent.com/u/10743879?v=3","gravatar_id":"","url":"https://api.github.com/users/wojtek-t","html_url":"https://github.com/wojtek-t","followers_url":"https://api.github.com/users/wojtek-t/followers","following_url":"https://api.github.com/users/wojtek-t/following{/other_user}","gists_url":"https://api.github.com/users/wojtek-t/gists{/gist_id}","starred_url":"https://api.github.com/users/wojtek-t/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wojtek-t/subscriptions","organizations_url":"https://api.github.com/users/wojtek-t/orgs","repos_url":"https://api.github.com/users/wojtek-t/repos","events_url":"https://api.github.com/users/wojtek-t/events{/privacy}","received_events_url":"https://api.github.com/users/wojtek-t/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/15","html_url":"https://github.com/kubernetes/kubernetes/milestones/v1.2","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/15/labels","id":1278499,"number":15,"title":"v1.2","description":"https://github.com/kubernetes/kubernetes/wiki/Release-1.2","creator":{"login":"bgrant0607","id":7725777,"avatar_url":"https://avatars.githubusercontent.com/u/7725777?v=3","gravatar_id":"","url":"https://api.github.com/users/bgrant0607","html_url":"https://github.com/bgrant0607","followers_url":"https://api.github.com/users/bgrant0607/followers","following_url":"https://api.github.com/users/bgrant0607/following{/other_user}","gists_url":"https://api.github.com/users/bgrant0607/gists{/gist_id}","starred_url":"https://api.github.com/users/bgrant0607/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bgrant0607/subscriptions","organizations_url":"https://api.github.com/users/bgrant0607/orgs","repos_url":"https://api.github.com/users/bgrant0607/repos","events_url":"https://api.github.com/users/bgrant0607/events{/privacy}","received_events_url":"https://api.github.com/users/bgrant0607/received_events","type":"User","site_admin":false},"open_issues":195,"closed_issues":272,"state":"open","created_at":"2015-08-29T03:54:16Z","updated_at":"2016-02-22T22:22:29Z","due_on":"2016-03-07T08:00:00Z","closed_at":null},"comments":24,"created_at":"2016-02-22T12:07:06Z","updated_at":"2016-02-22T23:07:05Z","closed_at":null,"body":"I've finally debugged the main problem that is causing load.go test failures in 1000-node clusters.\r\n\r\nThe problem is basically getPodController() method:\r\nhttps://github.com/kubernetes/kubernetes/blob/master/pkg/controller/replication/replication_controller.go#L202\r\n\r\nHere is exactly what is happening:\r\n- in load test in 1000-node cluster, we create ~3500 replication controllers, which altogether have 30.000 pods\r\n- then for each pod ReplicationControllerManager is creating a batch of pods:\r\nhttps://github.com/kubernetes/kubernetes/blob/master/pkg/controller/replication/replication_controller.go#L344\r\n- and then is waiting for them to be observed via watch\r\n- however, on top of watch, we have an Informer framework in ReplicationControllerManager:\r\nhttps://github.com/kubernetes/kubernetes/blob/master/pkg/controller/replication/replication_controller.go#L152\r\n- and the problem is that a single addPod/updatePod/deletePod call (which are called sequentially for every single add/update/delete pod action) is calling: getPodController:\r\nhttps://github.com/kubernetes/kubernetes/blob/master/pkg/controller/replication/replication_controller.go#L202\r\n- and getPodController has complexity O(#replicationControllers)\r\n\r\nWhich means that for any pod add/update/delete done in the system, ReplicationControllerManager is doing O(#replicationControllers) computations (which are btw non-trivial ones).\r\n\r\nThis is basically, why density.go (where we have a single RC) works pretty good, but load.go test basically doesn't work at all.\r\n\r\nIn my opinion this is a blocker for 1.2.\r\n\r\n@fgrzadkowski @gmarek @davidopp @lavalamp @kubernetes/goog-control-plane @kubernetes/sig-scalability "},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/187430299","html_url":"https://github.com/kubernetes/kubernetes/issues/21672#issuecomment-187430299","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/21672","id":187430299,"user":{"login":"bprashanth","id":10927820,"avatar_url":"https://avatars.githubusercontent.com/u/10927820?v=3","gravatar_id":"","url":"https://api.github.com/users/bprashanth","html_url":"https://github.com/bprashanth","followers_url":"https://api.github.com/users/bprashanth/followers","following_url":"https://api.github.com/users/bprashanth/following{/other_user}","gists_url":"https://api.github.com/users/bprashanth/gists{/gist_id}","starred_url":"https://api.github.com/users/bprashanth/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bprashanth/subscriptions","organizations_url":"https://api.github.com/users/bprashanth/orgs","repos_url":"https://api.github.com/users/bprashanth/repos","events_url":"https://api.github.com/users/bprashanth/events{/privacy}","received_events_url":"https://api.github.com/users/bprashanth/received_events","type":"User","site_admin":false},"created_at":"2016-02-22T23:07:05Z","updated_at":"2016-02-22T23:07:05Z","body":">I also vote for memoization. The more I think about it, the more I think adding parellization into the controller framework/informer is very dangerous. I don't think consumers of that code are expecting their handlers to be called in parallel right now.\r\n\r\n+1. Many controllers treat the serialization in the Informer like a write lock, including the Ingress controllers in contrib and who-knows-what downstream."}}