{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/61793","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/61793/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/61793/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/61793/events","html_url":"https://github.com/kubernetes/kubernetes/issues/61793","id":309134778,"number":61793,"title":"Kubelet has high CPU usage with high number of pods","user":{"login":"eugene-dounar","id":193521,"avatar_url":"https://avatars1.githubusercontent.com/u/193521?v=4","gravatar_id":"","url":"https://api.github.com/users/eugene-dounar","html_url":"https://github.com/eugene-dounar","followers_url":"https://api.github.com/users/eugene-dounar/followers","following_url":"https://api.github.com/users/eugene-dounar/following{/other_user}","gists_url":"https://api.github.com/users/eugene-dounar/gists{/gist_id}","starred_url":"https://api.github.com/users/eugene-dounar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eugene-dounar/subscriptions","organizations_url":"https://api.github.com/users/eugene-dounar/orgs","repos_url":"https://api.github.com/users/eugene-dounar/repos","events_url":"https://api.github.com/users/eugene-dounar/events{/privacy}","received_events_url":"https://api.github.com/users/eugene-dounar/received_events","type":"User","site_admin":false},"labels":[{"id":148734898,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/area/cadvisor","name":"area/cadvisor","color":"0052cc","default":false},{"id":116719829,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/area/kubelet","name":"area/kubelet","color":"0052cc","default":false},{"id":105146071,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/kind/bug","name":"kind/bug","color":"e11d21","default":false},{"id":173493665,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/node","name":"sig/node","color":"d2b48c","default":false}],"state":"closed","locked":false,"assignee":{"login":"dashpole","id":3262098,"avatar_url":"https://avatars1.githubusercontent.com/u/3262098?v=4","gravatar_id":"","url":"https://api.github.com/users/dashpole","html_url":"https://github.com/dashpole","followers_url":"https://api.github.com/users/dashpole/followers","following_url":"https://api.github.com/users/dashpole/following{/other_user}","gists_url":"https://api.github.com/users/dashpole/gists{/gist_id}","starred_url":"https://api.github.com/users/dashpole/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dashpole/subscriptions","organizations_url":"https://api.github.com/users/dashpole/orgs","repos_url":"https://api.github.com/users/dashpole/repos","events_url":"https://api.github.com/users/dashpole/events{/privacy}","received_events_url":"https://api.github.com/users/dashpole/received_events","type":"User","site_admin":false},"assignees":[{"login":"dashpole","id":3262098,"avatar_url":"https://avatars1.githubusercontent.com/u/3262098?v=4","gravatar_id":"","url":"https://api.github.com/users/dashpole","html_url":"https://github.com/dashpole","followers_url":"https://api.github.com/users/dashpole/followers","following_url":"https://api.github.com/users/dashpole/following{/other_user}","gists_url":"https://api.github.com/users/dashpole/gists{/gist_id}","starred_url":"https://api.github.com/users/dashpole/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dashpole/subscriptions","organizations_url":"https://api.github.com/users/dashpole/orgs","repos_url":"https://api.github.com/users/dashpole/repos","events_url":"https://api.github.com/users/dashpole/events{/privacy}","received_events_url":"https://api.github.com/users/dashpole/received_events","type":"User","site_admin":false}],"milestone":null,"comments":7,"created_at":"2018-03-27T21:09:06Z","updated_at":"2018-05-11T20:26:25Z","closed_at":"2018-05-11T20:23:22Z","author_association":"NONE","body":"**Is this a BUG REPORT or FEATURE REQUEST?**:\r\n/kind bug\r\n\r\n**What happened**:\r\nKubelet uses 70+ CPUs on 128 CPU instance when running 200+ pods\r\n\r\n**What you expected to happen**:\r\nKubelet uses under 10 CPUs\r\n\r\n**How to reproduce it (as minimally and precisely as possible)**:\r\n- Use AWS x1.32xlarge instance as Kubernetes node\r\n- Run 250 pods\r\n\r\n**Anything else we need to know?**:\r\nWe managed to run go pprof and bcc profiler. However the CPU usage pattern was irregular, and profiling data may be unrepresentative.\r\n\r\n[go pprof output](https://github.com/kubernetes/kubernetes/files/1853808/profile001.pdf)\r\n\r\nbcc-tools \"profile\" shows that kubelet spends most time starting `du`\r\n```\r\n    copy_pte_range\r\n    copy_page_range\r\n    copy_process.part.35\r\n    _do_fork\r\n    sys_clone\r\n    do_syscall_64\r\n    entry_SYSCALL_64_after_hwframe\r\n    --\r\n    syscall.RawSyscall6\r\n    syscall.forkExec\r\n    syscall.StartProcess\r\n    os.startProcess\r\n    os.StartProcess\r\n    os/exec.(*Cmd).Start\r\n    k8s.io/kubernetes/vendor/github.com/google/cadvisor/fs.GetDirDiskUsage\r\n    k8s.io/kubernetes/vendor/github.com/google/cadvisor/fs.(*RealFsInfo).GetDirDiskUsage\r\n    k8s.io/kubernetes/vendor/github.com/google/cadvisor/container/common.(*realFsHandler).update\r\n    k8s.io/kubernetes/vendor/github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage\r\n    runtime.goexit\r\n    -                kubelet (831528)\r\n        70\r\n```\r\n\r\n**Environment**:\r\n- Kubernetes version (use `kubectl version`):  v1.8.5\r\n- Cloud provider or hardware configuration: AWS x1.32xlarge\r\n- OS (e.g. from /etc/os-release): Ubuntu 16.04\r\n- Kernel (e.g. `uname -a`): `Linux REDACTED 4.15.3-041503-generic #201802120730` mainline Kernel\r\n- Install tools: kubespray\r\n"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/388476512","html_url":"https://github.com/kubernetes/kubernetes/issues/61793#issuecomment-388476512","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/61793","id":388476512,"user":{"login":"dashpole","id":3262098,"avatar_url":"https://avatars1.githubusercontent.com/u/3262098?v=4","gravatar_id":"","url":"https://api.github.com/users/dashpole","html_url":"https://github.com/dashpole","followers_url":"https://api.github.com/users/dashpole/followers","following_url":"https://api.github.com/users/dashpole/following{/other_user}","gists_url":"https://api.github.com/users/dashpole/gists{/gist_id}","starred_url":"https://api.github.com/users/dashpole/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dashpole/subscriptions","organizations_url":"https://api.github.com/users/dashpole/orgs","repos_url":"https://api.github.com/users/dashpole/repos","events_url":"https://api.github.com/users/dashpole/events{/privacy}","received_events_url":"https://api.github.com/users/dashpole/received_events","type":"User","site_admin":false},"created_at":"2018-05-11T20:26:25Z","updated_at":"2018-05-11T20:26:25Z","author_association":"CONTRIBUTOR","body":"Looks like overlay2 is fixed in 1.6.14, 1.7.11, and 1.8.? (https://github.com/kubernetes/kubernetes/pull/54882), and in 1.9+"}}
{"actor":{"display_login":"dashpole"}}