{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/63750","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/63750/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/63750/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/63750/events","html_url":"https://github.com/kubernetes/kubernetes/issues/63750","id":322512244,"node_id":"MDU6SXNzdWUzMjI1MTIyNDQ=","number":63750,"title":"Sub-optimal sizing of buckets exposed in the apiserver_request_latencies metrics","user":{"login":"mikkeloscar","id":128566,"node_id":"MDQ6VXNlcjEyODU2Ng==","avatar_url":"https://avatars0.githubusercontent.com/u/128566?v=4","gravatar_id":"","url":"https://api.github.com/users/mikkeloscar","html_url":"https://github.com/mikkeloscar","followers_url":"https://api.github.com/users/mikkeloscar/followers","following_url":"https://api.github.com/users/mikkeloscar/following{/other_user}","gists_url":"https://api.github.com/users/mikkeloscar/gists{/gist_id}","starred_url":"https://api.github.com/users/mikkeloscar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mikkeloscar/subscriptions","organizations_url":"https://api.github.com/users/mikkeloscar/orgs","repos_url":"https://api.github.com/users/mikkeloscar/repos","events_url":"https://api.github.com/users/mikkeloscar/events{/privacy}","received_events_url":"https://api.github.com/users/mikkeloscar/received_events","type":"User","site_admin":false},"labels":[{"id":105146071,"node_id":"MDU6TGFiZWwxMDUxNDYwNzE=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/kind/bug","name":"kind/bug","color":"e11d21","default":false},{"id":778118404,"node_id":"MDU6TGFiZWw3NzgxMTg0MDQ=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/lifecycle/stale","name":"lifecycle/stale","color":"795548","default":false},{"id":482207917,"node_id":"MDU6TGFiZWw0ODIyMDc5MTc=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/instrumentation","name":"sig/instrumentation","color":"d2b48c","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-05-12T13:39:05Z","updated_at":"2018-08-11T08:42:14Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"<!-- This form is for bug reports and feature requests ONLY!\r\n\r\nIf you're looking for help check [Stack Overflow](https://stackoverflow.com/questions/tagged/kubernetes) and the [troubleshooting guide](https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/).\r\n\r\nIf this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r\n-->\r\n\r\n**Is this a BUG REPORT or FEATURE REQUEST?**:\r\n\r\n/kind bug\r\n\r\n\r\n**What happened**:\r\n\r\nCurrently the buckets defined for the API server latency metric: `apiserver_request_latencies` are defined like this: https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go#L64-L65 meaning the smallest bucket is 125ms.\r\n\r\nIf you want to estimate the percentiles based on this then it only makes sense if the actual latency of your API server is around 125ms or higher otherwise you get quite skewed numbers.\r\n\r\nIn our case we are running more than 80 clusters on AWS on different instance types and out of those 80 clusters only 3 has a latency higher than 125ms (for verb=\"GET\") as reported clientside in the `apiserver_request_latencies_summary` metric. The rest has a latency below 100ms and most of them even a latency under 10ms (Looking at GET p99).\r\n\r\nHowever, because of the bucket sizes all latencies are reported to be close to or higher than 125ms if we try to estimate it with a query like this:\r\n\r\n```\r\nhistogram_quantile(0.99, sum(rate(apiserver_request_latencies_bucket[5m])) by (le, verb))\r\n```\r\n\r\n![p99](https://user-images.githubusercontent.com/128566/39957809-4b28b548-55f9-11e8-9f8e-5592468a5470.png)\r\n\r\nIt's clear that the estimated values are much higher than what is reported clientside in the `apiserver_request_latencies_summary` metric.\r\n\r\nLooking at the aggregated estimations thus gives the wrong impression of how the service is performing.\r\n\r\n**What you expected to happen**:\r\n\r\nI realize that it's impossible to pick bucket sizes that will be a perfect fit for everyone since the latency ofc. will heavily depend on the load of the clusters and the backing hardward. However, I think it would be more sensible to pick buckets that start below 125ms.\r\n\r\nThe prometheus library provides some default bucket sizes: https://github.com/prometheus/client_golang/blob/master/prometheus/histogram.go#L59\r\n\r\n> The default buckets are tailored to broadly measure the response time (in seconds) of a network service.\r\n\r\nThese buckets are in the range from 5ms to 10s which seems much more sensible to cover a wide range of Kubernetes deployments.\r\n\r\nI tried to change the buckets used and this is what I saw:\r\n\r\nQuery: \r\n\r\n```\r\nhistogram_quantile(0.99, sum(rate(apiserver_request_latencies_bucket[5m])) by (le, verb))\r\n```\r\n\r\nResult with the current buckets:\r\n\r\n```\r\n{verb=\"DELETE\"}\t123750\r\n{verb=\"GET\"}\t123750\r\n{verb=\"LIST\"}\t123875.25303643724\r\n{verb=\"PATCH\"}\t123750\r\n{verb=\"POST\"}\t123750.00000000001\r\n{verb=\"PUT\"}\t123750\r\n{verb=\"WATCH\"}\t8000000\r\n```\r\n\r\nResult with the default prometheus bucket sizes:\r\n\r\n```\r\n{verb=\"DELETE\"}\tNaN\r\n{verb=\"GET\"}\t4985.64\r\n{verb=\"LIST\"}\t23381.395348837177\r\n{verb=\"PATCH\"}\t19100.000000000007\r\n{verb=\"POST\"}\t4950\r\n{verb=\"PUT\"}\t9892.028985507248\r\n{verb=\"WATCH\"}\t10000000\r\n```\r\n\r\nValues are in microseconds.\r\n\r\nBased on this I suggest we change the buckets defined for the metric to be more useful. I'm happy to provide a PR if you agree with this.\r\n\r\n**How to reproduce it (as minimally and precisely as possible)**:\r\n\r\nScrape the kube-apiserver prometheus metrics and query them like this:\r\n\r\n```\r\nhistogram_quantile(0.99, sum(rate(apiserver_request_latencies_bucket[5m])) by (le, verb))\r\n```\r\n\r\n\r\n**Anything else we need to know?**:\r\n\r\n**Environment**: Not relevant.\r\n- Kubernetes version (use `kubectl version`):\r\n- Cloud provider or hardware configuration:\r\n- OS (e.g. from /etc/os-release):\r\n- Kernel (e.g. `uname -a`):\r\n- Install tools:\r\n- Others:\r\n\r\n/sig instrumentation"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/412261132","html_url":"https://github.com/kubernetes/kubernetes/issues/63750#issuecomment-412261132","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/63750","id":412261132,"node_id":"MDEyOklzc3VlQ29tbWVudDQxMjI2MTEzMg==","user":{"login":"nikhita","id":16105680,"node_id":"MDQ6VXNlcjE2MTA1Njgw","avatar_url":"https://avatars1.githubusercontent.com/u/16105680?v=4","gravatar_id":"","url":"https://api.github.com/users/nikhita","html_url":"https://github.com/nikhita","followers_url":"https://api.github.com/users/nikhita/followers","following_url":"https://api.github.com/users/nikhita/following{/other_user}","gists_url":"https://api.github.com/users/nikhita/gists{/gist_id}","starred_url":"https://api.github.com/users/nikhita/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nikhita/subscriptions","organizations_url":"https://api.github.com/users/nikhita/orgs","repos_url":"https://api.github.com/users/nikhita/repos","events_url":"https://api.github.com/users/nikhita/events{/privacy}","received_events_url":"https://api.github.com/users/nikhita/received_events","type":"User","site_admin":false},"created_at":"2018-08-11T08:42:14Z","updated_at":"2018-08-11T08:42:14Z","author_association":"MEMBER","body":"/remove-lifecycle stale"}}
{"actor":{"display_login":"nikhita"}}