{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/19341","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/19341/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/19341/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/19341/events","html_url":"https://github.com/kubernetes/kubernetes/issues/19341","id":125251005,"number":19341,"title":"reduce kubelet overhead","user":{"login":"timothysc","id":169553,"avatar_url":"https://avatars.githubusercontent.com/u/169553?v=3","gravatar_id":"","url":"https://api.github.com/users/timothysc","html_url":"https://github.com/timothysc","followers_url":"https://api.github.com/users/timothysc/followers","following_url":"https://api.github.com/users/timothysc/following{/other_user}","gists_url":"https://api.github.com/users/timothysc/gists{/gist_id}","starred_url":"https://api.github.com/users/timothysc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/timothysc/subscriptions","organizations_url":"https://api.github.com/users/timothysc/orgs","repos_url":"https://api.github.com/users/timothysc/repos","events_url":"https://api.github.com/users/timothysc/events{/privacy}","received_events_url":"https://api.github.com/users/timothysc/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/area/performance","name":"area/performance","color":"d4c5f9"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/node","name":"team/node","color":"d2b48c"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-01-06T20:01:26Z","updated_at":"2016-01-06T21:51:28Z","closed_at":null,"body":"**Stack configuration:**\r\nfedora 23 \r\nk8's tip-of-master ee006e767ff1d435c525c4c11547e06d364e7621\r\netcd 2.2.1 \r\ndocker 1.9.1 \r\n\r\n**experiment:**\r\n1. mock local-up-cluster.sh to allow --max-pods=\"100\" and start local cluster\r\n2. create a simple replication controller 100 wide that starts gcr.io/google_containers/pause:2.0 container with no resource limits specified. \r\n3. during flux + steady state measure kubelet memory using smap data: https://raw.githubusercontent.com/pixelb/ps_mem/master/ps_mem.py, and watch cpu consumption with 'perf top' \r\n\r\n**observed (steady state):**\r\n```\r\n$ sudo ./ps_mem.py -p 17978\r\n Private  +   Shared  =  RAM used\tProgram\r\n 93.9 MiB +   1.2 MiB =  95.1 MiB\tdocker\r\n\r\n$ sudo ./ps_mem.py -p 18313\r\n Private  +   Shared  =  RAM used\tProgram\r\n552.5 MiB + 259.5 KiB = 552.8 MiB\tkubelet\r\n```\r\ncpu spikes occur for >2 cores on the kubelet as well. \r\n\r\n/cc @kubernetes/sig-node @kubernetes/sig-scalability "},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/169474065","html_url":"https://github.com/kubernetes/kubernetes/issues/19341#issuecomment-169474065","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/19341","id":169474065,"user":{"login":"timothysc","id":169553,"avatar_url":"https://avatars.githubusercontent.com/u/169553?v=3","gravatar_id":"","url":"https://api.github.com/users/timothysc","html_url":"https://github.com/timothysc","followers_url":"https://api.github.com/users/timothysc/followers","following_url":"https://api.github.com/users/timothysc/following{/other_user}","gists_url":"https://api.github.com/users/timothysc/gists{/gist_id}","starred_url":"https://api.github.com/users/timothysc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/timothysc/subscriptions","organizations_url":"https://api.github.com/users/timothysc/orgs","repos_url":"https://api.github.com/users/timothysc/repos","events_url":"https://api.github.com/users/timothysc/events{/privacy}","received_events_url":"https://api.github.com/users/timothysc/received_events","type":"User","site_admin":false},"created_at":"2016-01-06T21:51:28Z","updated_at":"2016-01-06T21:51:28Z","body":"That is very odd that we have such a large difference, what version of golang are you running? \r\n"}}