{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kube-state-metrics/issues/534","repository_url":"https://api.github.com/repos/kubernetes/kube-state-metrics","labels_url":"https://api.github.com/repos/kubernetes/kube-state-metrics/issues/534/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kube-state-metrics/issues/534/comments","events_url":"https://api.github.com/repos/kubernetes/kube-state-metrics/issues/534/events","html_url":"https://github.com/kubernetes/kube-state-metrics/pull/534","id":356522118,"node_id":"MDExOlB1bGxSZXF1ZXN0MjEyNzYxNDg5","number":534,"title":" pkg/collectors: Introduce MetricsStore updated by Reflectors","user":{"login":"mxinden","id":7047859,"node_id":"MDQ6VXNlcjcwNDc4NTk=","avatar_url":"https://avatars0.githubusercontent.com/u/7047859?v=4","gravatar_id":"","url":"https://api.github.com/users/mxinden","html_url":"https://github.com/mxinden","followers_url":"https://api.github.com/users/mxinden/followers","following_url":"https://api.github.com/users/mxinden/following{/other_user}","gists_url":"https://api.github.com/users/mxinden/gists{/gist_id}","starred_url":"https://api.github.com/users/mxinden/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mxinden/subscriptions","organizations_url":"https://api.github.com/users/mxinden/orgs","repos_url":"https://api.github.com/users/mxinden/repos","events_url":"https://api.github.com/users/mxinden/events{/privacy}","received_events_url":"https://api.github.com/users/mxinden/received_events","type":"User","site_admin":false},"labels":[{"id":489138866,"node_id":"MDU6TGFiZWw0ODkxMzg4NjY=","url":"https://api.github.com/repos/kubernetes/kube-state-metrics/labels/cncf-cla:%20no","name":"cncf-cla: no","color":"e11d21","default":false},{"id":704084945,"node_id":"MDU6TGFiZWw3MDQwODQ5NDU=","url":"https://api.github.com/repos/kubernetes/kube-state-metrics/labels/size/XXL","name":"size/XXL","color":"ee0000","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":12,"created_at":"2018-09-03T14:13:26Z","updated_at":"2018-10-03T08:11:02Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/kubernetes/kube-state-metrics/pulls/534","html_url":"https://github.com/kubernetes/kube-state-metrics/pull/534","diff_url":"https://github.com/kubernetes/kube-state-metrics/pull/534.diff","patch_url":"https://github.com/kubernetes/kube-state-metrics/pull/534.patch"},"body":"**What this PR does / why we need it**:\r\n\r\nAs discussed in #498 kube-state-metrics faces the following two performance\r\nissues when deployed in bigger Kubernetes environments:\r\n\r\n1. Long response time on metrics scrape\r\n2. High memory usage\r\n\r\nThis patch introduces a custom client go store, MetricsStore. It is updated by a\r\ncorresponding reflector. Instead of generating the Prometheus custom metrics on\r\ndemand, metrics are generated on new Kubernetes object arrival.\r\n\r\nThis pull request represents an early *work-in-progress* state to enable early\r\nfeedback. It targets the *performance-optimizations* branch, instead of *master*.\r\n\r\nKnown ToDos:\r\n\r\n- Re-enable multi namespace support\r\n- Re-enable white/black listing\r\n- Adjust remaining `collectors/*_test.go`\r\n\r\n**Premature benchmarking results**:\r\n\r\n1000 configmaps, 1000 pods, 100 scrapes:\r\n\r\n\r\n\r\n\r\n\r\n```shell\r\n$ git checkout master\r\n$ go test -bench .                    \r\ngoos: linux\r\ngoarch: amd64\r\npkg: k8s.io/kube-state-metrics\r\nBenchmarkKubeStateMetrics-4            1        10267655552 ns/op\r\nPASS\r\nok      k8s.io/kube-state-metrics       10.282s\r\n\r\n```\r\n\r\n[master_cpu.pdf](https://github.com/kubernetes/kube-state-metrics/files/2345809/master_cpu.pdf)\r\n[master_mem.pdf](https://github.com/kubernetes/kube-state-metrics/files/2345810/master_mem.pdf)\r\n\r\n```shell\r\n$ git checkout metricsstore\r\n$ go test -bench .\r\ngoos: linux\r\ngoarch: amd64\r\npkg: k8s.io/kube-state-metrics\r\nBenchmarkKubeStateMetrics-4            1        1876562818 ns/op\r\n--- BENCH: BenchmarkKubeStateMetrics-4\r\n        main_test.go:39: starting kube-state-metrics benchmark with fixtureMultiplier 1000 and requestCount 100\r\nPASS\r\nok      k8s.io/kube-state-metrics       1.892s\r\n```\r\n\r\n[metricsstore_cpu.pdf](https://github.com/kubernetes/kube-state-metrics/files/2345825/metricsstore_cpu.pdf)\r\n[metricsstore_mem.pdf](https://github.com/kubernetes/kube-state-metrics/files/2345826/metricsstore_mem.pdf)\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n**Which issue(s) this PR fixes**:\r\n\r\nRelates to https://github.com/kubernetes/kube-state-metrics/issues/498\r\n\r\n\r\n//CC @metalmatze @brancz @squat @s-urbaniak\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/kubernetes/kube-state-metrics/issues/comments/426548823","html_url":"https://github.com/kubernetes/kube-state-metrics/pull/534#issuecomment-426548823","issue_url":"https://api.github.com/repos/kubernetes/kube-state-metrics/issues/534","id":426548823,"node_id":"MDEyOklzc3VlQ29tbWVudDQyNjU0ODgyMw==","user":{"login":"mxinden","id":7047859,"node_id":"MDQ6VXNlcjcwNDc4NTk=","avatar_url":"https://avatars0.githubusercontent.com/u/7047859?v=4","gravatar_id":"","url":"https://api.github.com/users/mxinden","html_url":"https://github.com/mxinden","followers_url":"https://api.github.com/users/mxinden/followers","following_url":"https://api.github.com/users/mxinden/following{/other_user}","gists_url":"https://api.github.com/users/mxinden/gists{/gist_id}","starred_url":"https://api.github.com/users/mxinden/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mxinden/subscriptions","organizations_url":"https://api.github.com/users/mxinden/orgs","repos_url":"https://api.github.com/users/mxinden/repos","events_url":"https://api.github.com/users/mxinden/events{/privacy}","received_events_url":"https://api.github.com/users/mxinden/received_events","type":"User","site_admin":false},"created_at":"2018-10-03T08:11:02Z","updated_at":"2018-10-03T08:11:02Z","author_association":"CONTRIBUTOR","body":"Thanks a lot @ehashman for running this in a bigger environment. Those numbers look great.\r\n\r\n> responses from KSM are no longer gzipped\r\n\r\nRight, I will look into that. Thanks for the hint.\r\n\r\nI have pushed a bugfix since, here are the updated images:\r\n- quay.io/mxinden/kube-state-metrics:v1.4.0-perf.8\r\n- minden/kube-state-metrics:v1.4.0-perf.8\r\n\r\nIn case you have a chance, would you mind trying those out as well?\r\n\r\n---\r\n\r\nOur scalability team gave me access to a big cluster (~250 nodes) with some artificial workload on it. Next to the above mentioned improvements by @ehashman scrape request duration also improved and is a lot more stable now (top=v1.4.0; bottom=v1.4.0-perf.8):\r\n\r\n![scrape_duration](https://user-images.githubusercontent.com/7047859/46398099-71e5b700-c6f4-11e8-9187-80157178cef5.jpg)\r\n\r\n"}}
{"actor":{"display_login":"mxinden"}}