{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/27456","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/27456/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/27456/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/27456/events","html_url":"https://github.com/kubernetes/kubernetes/issues/27456","id":160490455,"number":27456,"title":"[k8s.io] Kubelet [Serial] [Slow] [k8s.io] regular resource usage tracking resource tracking","user":{"login":"krousey","id":157083,"avatar_url":"https://avatars.githubusercontent.com/u/157083?v=3","gravatar_id":"","url":"https://api.github.com/users/krousey","html_url":"https://github.com/krousey","followers_url":"https://api.github.com/users/krousey/followers","following_url":"https://api.github.com/users/krousey/following{/other_user}","gists_url":"https://api.github.com/users/krousey/gists{/gist_id}","starred_url":"https://api.github.com/users/krousey/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/krousey/subscriptions","organizations_url":"https://api.github.com/users/krousey/orgs","repos_url":"https://api.github.com/users/krousey/repos","events_url":"https://api.github.com/users/krousey/events{/privacy}","received_events_url":"https://api.github.com/users/krousey/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/kind/flake","name":"kind/flake","color":"f7c6c7"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/priority/P1","name":"priority/P1","color":"eb6420"}],"state":"open","locked":false,"assignee":{"login":"yujuhong","id":10524058,"avatar_url":"https://avatars.githubusercontent.com/u/10524058?v=3","gravatar_id":"","url":"https://api.github.com/users/yujuhong","html_url":"https://github.com/yujuhong","followers_url":"https://api.github.com/users/yujuhong/followers","following_url":"https://api.github.com/users/yujuhong/following{/other_user}","gists_url":"https://api.github.com/users/yujuhong/gists{/gist_id}","starred_url":"https://api.github.com/users/yujuhong/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yujuhong/subscriptions","organizations_url":"https://api.github.com/users/yujuhong/orgs","repos_url":"https://api.github.com/users/yujuhong/repos","events_url":"https://api.github.com/users/yujuhong/events{/privacy}","received_events_url":"https://api.github.com/users/yujuhong/received_events","type":"User","site_admin":false},"milestone":null,"comments":9,"created_at":"2016-06-15T18:23:49Z","updated_at":"2016-06-18T01:36:29Z","closed_at":null,"body":"https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-soak-continuous-e2e-gke/7477\r\n\r\nI'm filing these all together since they seem to be related.\r\n\r\n[k8s.io] Kubelet [Serial] [Slow] [k8s.io] regular resource usage tracking resource tracking for 0 pods per node\r\n\r\n```\r\n/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubelet_perf.go:277\r\nJun 14 03:02:42.724: Memory usage exceeding limits:\r\n node gke-jenkins-e2e-default-pool-91767c58-01c5:\r\n container \"runtime\": expected RSS memory (MB) < 89128960; got 96772096, container \"kubelet\": expected RSS memory (MB) < 73400320; got 92086272\r\nnode gke-jenkins-e2e-default-pool-91767c58-0xjm:\r\n container \"kubelet\": expected RSS memory (MB) < 73400320; got 112783360, container \"runtime\": expected RSS memory (MB) < 89128960; got 114671616\r\nnode gke-jenkins-e2e-default-pool-91767c58-vpv2:\r\n container \"kubelet\": expected RSS memory (MB) < 73400320; got 116867072, container \"runtime\": expected RSS memory (MB) < 89128960; got 111587328\r\n```\r\n\r\n[k8s.io] Kubelet [Serial] [Slow] [k8s.io] regular resource usage tracking resource tracking for 35 pods per node\r\n\r\n```\r\n/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubelet_perf.go:277\r\nJun 14 03:35:44.460: Memory usage exceeding limits:\r\n node gke-jenkins-e2e-default-pool-91767c58-01c5:\r\n container \"kubelet\": expected RSS memory (MB) < 73400320; got 101568512\r\nnode gke-jenkins-e2e-default-pool-91767c58-0xjm:\r\n container \"kubelet\": expected RSS memory (MB) < 73400320; got 122929152\r\nnode gke-jenkins-e2e-default-pool-91767c58-vpv2:\r\n container \"kubelet\": expected RSS memory (MB) < 73400320; got 126533632\r\n```\r\n\r\n[k8s.io] Kubelet [Serial] [Slow] [k8s.io] regular resource usage tracking resource tracking for 100 pods per node\r\n\r\n```\r\n/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubelet_perf.go:277\r\nJun 14 04:57:48.807: Memory usage exceeding limits:\r\n node gke-jenkins-e2e-default-pool-91767c58-01c5:\r\n container \"kubelet\": expected RSS memory (MB) < 83886080; got 118796288\r\nnode gke-jenkins-e2e-default-pool-91767c58-0xjm:\r\n container \"kubelet\": expected RSS memory (MB) < 83886080; got 139313152\r\nnode gke-jenkins-e2e-default-pool-91767c58-vpv2:\r\n container \"kubelet\": expected RSS memory (MB) < 83886080; got 136806400\r\n```\r\n\r\n#27374"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/226914467","html_url":"https://github.com/kubernetes/kubernetes/issues/27456#issuecomment-226914467","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/27456","id":226914467,"user":{"login":"yujuhong","id":10524058,"avatar_url":"https://avatars.githubusercontent.com/u/10524058?v=3","gravatar_id":"","url":"https://api.github.com/users/yujuhong","html_url":"https://github.com/yujuhong","followers_url":"https://api.github.com/users/yujuhong/followers","following_url":"https://api.github.com/users/yujuhong/following{/other_user}","gists_url":"https://api.github.com/users/yujuhong/gists{/gist_id}","starred_url":"https://api.github.com/users/yujuhong/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yujuhong/subscriptions","organizations_url":"https://api.github.com/users/yujuhong/orgs","repos_url":"https://api.github.com/users/yujuhong/repos","events_url":"https://api.github.com/users/yujuhong/events{/privacy}","received_events_url":"https://api.github.com/users/yujuhong/received_events","type":"User","site_admin":false},"created_at":"2016-06-18T01:36:28Z","updated_at":"2016-06-18T01:36:28Z","body":"I did a few things in my cluster to reproduce this issue:\r\n - adjusted the memory profile rate to get a more accurate picture (at the cost of performance, which I don't care)\r\n - tampered with the inotify code to increase the `inotify.Event` size (*100x*) to amplify the impact of these objects.\r\n\r\nAfter kubelet was up for ~1 hour, and has created/deleted hundreds of pods, the heap profile showed a significant usage from of `readEvents()`\r\n\r\n```\r\n(pprof) top\r\n69.38MB of 78.38MB total (88.52%)\r\nDropped 2430 nodes (cum <= 0.39MB)\r\nShowing top 10 nodes out of 149 (cum >= 0.58MB)\r\n      flat  flat%   sum%        cum   cum%\r\n   59.67MB 76.12% 76.12%    59.67MB 76.12%  k8s.io/kubernetes/vendor/golang.org/x/exp/inotify.(*Watcher).readEvents\r\n    2.28MB  2.90% 79.03%     2.28MB  2.90%  bytes.(*Buffer).WriteTo\r\n    1.84MB  2.34% 81.37%     1.84MB  2.34%  k8s.io/kubernetes/vendor/github.com/beorn7/perks/quantile.Samples.Swap\r\n    1.07MB  1.36% 82.73%     1.07MB  1.36%  k8s.io/kubernetes/vendor/github.com/beorn7/perks/quantile.(*stream).merge\r\n    0.98MB  1.25% 83.97%     0.98MB  1.25%  strings.Join\r\n    0.89MB  1.14% 85.11%     1.32MB  1.69%  k8s.io/kubernetes/vendor/github.com/google/cadvisor/container/libcontainer.toContainerStats3\r\n    0.82MB  1.04% 86.16%     0.82MB  1.04%  k8s.io/kubernetes/vendor/github.com/prometheus/client_golang/prometheus.(*registry).giveMetric\r\n    0.65MB  0.83% 86.99%     0.65MB  0.83%  encoding/json.(*decodeState).literalStore\r\n    0.63MB   0.8% 87.78%     0.63MB   0.8%  regexp.(*bitState).reset\r\n    0.58MB  0.74% 88.52%     0.58MB  0.75%  reflect.Value.call\r\n```\r\n\r\nThe `inotify.Event` object is last used in [processEvent()](https://github.com/kubernetes/kubernetes/blob/237f90d6ee159ac5fb5dd4d2b1a814b23a68ffd9/vendor/github.com/google/cadvisor/manager/watcher/raw/raw.go#L148). \r\nOther than that, the `inotify.Event` objects don't seem to be referenced anywhere. Not sure why GC didn't reclaim them at all."}}
{"actor":{"display_login":"yujuhong"}}