{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/16943","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/16943/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/16943/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/16943/events","html_url":"https://github.com/kubernetes/kubernetes/issues/16943","id":115580057,"number":16943,"title":"RFC: Determine the scalability goal for v1.2 kubelet","user":{"login":"yujuhong","id":10524058,"avatar_url":"https://avatars.githubusercontent.com/u/10524058?v=3","gravatar_id":"","url":"https://api.github.com/users/yujuhong","html_url":"https://github.com/yujuhong","followers_url":"https://api.github.com/users/yujuhong/followers","following_url":"https://api.github.com/users/yujuhong/following{/other_user}","gists_url":"https://api.github.com/users/yujuhong/gists{/gist_id}","starred_url":"https://api.github.com/users/yujuhong/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yujuhong/subscriptions","organizations_url":"https://api.github.com/users/yujuhong/orgs","repos_url":"https://api.github.com/users/yujuhong/repos","events_url":"https://api.github.com/users/yujuhong/events{/privacy}","received_events_url":"https://api.github.com/users/yujuhong/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/priority/P1","name":"priority/P1","color":"eb6420"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/node","name":"team/node","color":"d2b48c"}],"state":"open","locked":false,"assignee":{"login":"yujuhong","id":10524058,"avatar_url":"https://avatars.githubusercontent.com/u/10524058?v=3","gravatar_id":"","url":"https://api.github.com/users/yujuhong","html_url":"https://github.com/yujuhong","followers_url":"https://api.github.com/users/yujuhong/followers","following_url":"https://api.github.com/users/yujuhong/following{/other_user}","gists_url":"https://api.github.com/users/yujuhong/gists{/gist_id}","starred_url":"https://api.github.com/users/yujuhong/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yujuhong/subscriptions","organizations_url":"https://api.github.com/users/yujuhong/orgs","repos_url":"https://api.github.com/users/yujuhong/repos","events_url":"https://api.github.com/users/yujuhong/events{/privacy}","received_events_url":"https://api.github.com/users/yujuhong/received_events","type":"User","site_admin":false},"milestone":null,"comments":30,"created_at":"2015-11-06T20:21:54Z","updated_at":"2016-01-21T01:12:01Z","closed_at":null,"body":"The umbrella issue for kubelet's scalability: #12540\r\n\r\nKubelet manages and monitors all user pods on a node to ensure their container states meet the desired pod specifications. Every pod and/or container Kubelet manages incurs a certain resource overhead even in the absence of active events (i.e., no spec change or container lifecycle events). The exact per-pod overhead varies based on the detailed pod specification, but in general, such steady-state management overhead of is proportional to the number of active pods/containers. When there no active pods, the resource overhead should be negligible.\r\n\r\nBelow are some resource usage numbers for the v1.1 kubelet.\r\n\r\n## v1.1 Resource Usage** (commit: dd1187c3ff9a5)\r\n\r\nCluster setup/workload:\r\n - Three n1-standard-2 nodes on GCE (i.e., 2-core)\r\n - Running all default cluster add-on pods\r\n - 0 and 40 single-container pause pods per node\r\n - 30-minutes measurement period\r\n - The cpu usage numbers are aggregated by ~~10s~~ ***1s*** intervals\r\n - The working set memory size was hard to measured correctly (see #12422 for details). In the experiments, I set logging level to 0 and lowered the cgroup memory limit of kubelet and docker to create a constant memory pressure, then let them bounce back until the usage was stable. The numbers are approximates and should only be considered as ballparks.\r\n\r\n### 0 pause pods per node\r\n\r\n|  | median cpu | 95th% cpu | memory |\r\n| --------- | ------- | ------ | ------- |\r\n| kubelet | 0.02 | 0.05 | 35MB |\r\n| docker | 0.01 | 0.03 | 35MB |\r\n| kube-proxy| <0.001 | 0.01 | <5MB* |\r\n\r\n### 40 pause pods per node\r\n\r\n|  | median cpu | 95th% cpu | memory |\r\n| --------- | ------- | ------ | ------- |\r\n| kubelet | 0.18 | 0.64 | 55MB |\r\n| docker | 0.01 | 0.62 | 70MB |\r\n| kube-proxy | <0.001 | 0.03 | <5MB* |\r\n\r\n*kube-proxy’s memory usage depends on the number of services in the cluster. In this test case, the number of services is 7.*\r\n\r\nNote that, kubelet syncs all pods periodically, hence even though the median cpu usage tends to be lower, the cpu spikes are significant.\r\n### 90 pause pods per node?\r\n\r\nv1.1 does not support 90 pods per node, but I did some rough measurement and the median cpu usage of kubelet and docker combined varies from 0.35 to 1.2 cores -- it was right on the edge of a steep curve. Beyond 70th%, it was consistently > 1.4 core.\r\n\r\n\r\n### Reaction latency\r\nThe latency Kubelet takes to notice a container event (e.g. died) varies and could be up to **10+** seconds, since the sync period is 10s.\r\n\r\n## v1.2 Resource Usage Target\r\n\r\nFor v1.2  we want to set a goal so that we can work towards to it. The goal would affect what approach we pick to tackle the issues.\r\n\r\nBelow are some tentative targets. Suggestions/comments are welcome.\r\n- Number of pods: 90 pause pods + all add-on pods\r\n- Reaction latency (for majority of the container evens): within 3s\r\n - This is somewhat arbitrary. We have a pretty long reaction latency today, and we can certainly lower this more *if* we adopt the docker event stream (#16831). What's an acceptable value?\r\n- cpu usage for kubelet and docker combined:\r\n  - median: 0.2 core (10% of the 2-core machine)\r\n  - 95th%: 0.6 core\r\n  - 99th%: TBD\r\n \r\n\r\n## Caveats\r\n - The detailed pod spec may affect the resource usage. For example, if every container defines a health check, the overhead of periodic probing may significantly increase the cpu usage. We should also publish the usage numbers of the worst-case scenario.\r\n - We focus on the steady-state resource usage. When there are non-monitoring work involved (e.g. container creation/deletion), the cpu usage would of course increase. We can further restrict the usage by either rate-limiting operations, or set a reasonable cpu share for kubelet and docker. Again, this is not the focus of the issue.\r\n - The built-in cadvisor constantly monitors all containers stats. The cpu usage of cadvisor for 100 pods is not ideal (see https://github.com/kubernetes/kubernetes/issues/16296#issuecomment-152365160), but seems tolerable at the moment. We will optimize that if it becomes the bottleneck.\r\n - We will want to optimize the 99th% cpu usage later as well. The current measurements are not very meaningful because kubelet and docker each consumes ~1 core at 99th% cpu usage.\r\n - If we use a smaller cpu measurement interval, we might be able to capture the fine-grained, more dynamic behavior. On the other hand, the measurement overhead would increase.\r\n - The goal will be specifically for docker, but the most of the improvement will apply to rkt as well.\r\n\r\n/cc @kubernetes/goog-node "},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/173419007","html_url":"https://github.com/kubernetes/kubernetes/issues/16943#issuecomment-173419007","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/16943","id":173419007,"user":{"login":"yujuhong","id":10524058,"avatar_url":"https://avatars.githubusercontent.com/u/10524058?v=3","gravatar_id":"","url":"https://api.github.com/users/yujuhong","html_url":"https://github.com/yujuhong","followers_url":"https://api.github.com/users/yujuhong/followers","following_url":"https://api.github.com/users/yujuhong/following{/other_user}","gists_url":"https://api.github.com/users/yujuhong/gists{/gist_id}","starred_url":"https://api.github.com/users/yujuhong/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yujuhong/subscriptions","organizations_url":"https://api.github.com/users/yujuhong/orgs","repos_url":"https://api.github.com/users/yujuhong/repos","events_url":"https://api.github.com/users/yujuhong/events{/privacy}","received_events_url":"https://api.github.com/users/yujuhong/received_events","type":"User","site_admin":false},"created_at":"2016-01-21T01:12:01Z","updated_at":"2016-01-21T01:12:01Z","body":"> We should re-update and compare numbers once #19850 and accompanying bugz are fixed.\r\n\r\nYes. Hopefully it won't take long..."}}