{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/51135","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/51135/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/51135/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/51135/events","html_url":"https://github.com/kubernetes/kubernetes/issues/51135","id":252047593,"number":51135,"title":"Avoid setting CPU limits for Guaranteed pods","user":{"login":"vishh","id":7063592,"avatar_url":"https://avatars2.githubusercontent.com/u/7063592?v=4","gravatar_id":"","url":"https://api.github.com/users/vishh","html_url":"https://github.com/vishh","followers_url":"https://api.github.com/users/vishh/followers","following_url":"https://api.github.com/users/vishh/following{/other_user}","gists_url":"https://api.github.com/users/vishh/gists{/gist_id}","starred_url":"https://api.github.com/users/vishh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vishh/subscriptions","organizations_url":"https://api.github.com/users/vishh/orgs","repos_url":"https://api.github.com/users/vishh/repos","events_url":"https://api.github.com/users/vishh/events{/privacy}","received_events_url":"https://api.github.com/users/vishh/received_events","type":"User","site_admin":false},"labels":[{"id":105146071,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/kind/bug","name":"kind/bug","color":"e11d21","default":false},{"id":267761362,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/kind/feature","name":"kind/feature","color":"c7def8","default":false},{"id":114528223,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/priority/important-soon","name":"priority/important-soon","color":"eb6420","default":false},{"id":173493665,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/node","name":"sig/node","color":"d2b48c","default":false}],"state":"open","locked":false,"assignee":{"login":"derekwaynecarr","id":6233452,"avatar_url":"https://avatars3.githubusercontent.com/u/6233452?v=4","gravatar_id":"","url":"https://api.github.com/users/derekwaynecarr","html_url":"https://github.com/derekwaynecarr","followers_url":"https://api.github.com/users/derekwaynecarr/followers","following_url":"https://api.github.com/users/derekwaynecarr/following{/other_user}","gists_url":"https://api.github.com/users/derekwaynecarr/gists{/gist_id}","starred_url":"https://api.github.com/users/derekwaynecarr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/derekwaynecarr/subscriptions","organizations_url":"https://api.github.com/users/derekwaynecarr/orgs","repos_url":"https://api.github.com/users/derekwaynecarr/repos","events_url":"https://api.github.com/users/derekwaynecarr/events{/privacy}","received_events_url":"https://api.github.com/users/derekwaynecarr/received_events","type":"User","site_admin":false},"assignees":[{"login":"derekwaynecarr","id":6233452,"avatar_url":"https://avatars3.githubusercontent.com/u/6233452?v=4","gravatar_id":"","url":"https://api.github.com/users/derekwaynecarr","html_url":"https://github.com/derekwaynecarr","followers_url":"https://api.github.com/users/derekwaynecarr/followers","following_url":"https://api.github.com/users/derekwaynecarr/following{/other_user}","gists_url":"https://api.github.com/users/derekwaynecarr/gists{/gist_id}","starred_url":"https://api.github.com/users/derekwaynecarr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/derekwaynecarr/subscriptions","organizations_url":"https://api.github.com/users/derekwaynecarr/orgs","repos_url":"https://api.github.com/users/derekwaynecarr/repos","events_url":"https://api.github.com/users/derekwaynecarr/events{/privacy}","received_events_url":"https://api.github.com/users/derekwaynecarr/received_events","type":"User","site_admin":false},{"login":"vishh","id":7063592,"avatar_url":"https://avatars2.githubusercontent.com/u/7063592?v=4","gravatar_id":"","url":"https://api.github.com/users/vishh","html_url":"https://github.com/vishh","followers_url":"https://api.github.com/users/vishh/followers","following_url":"https://api.github.com/users/vishh/following{/other_user}","gists_url":"https://api.github.com/users/vishh/gists{/gist_id}","starred_url":"https://api.github.com/users/vishh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vishh/subscriptions","organizations_url":"https://api.github.com/users/vishh/orgs","repos_url":"https://api.github.com/users/vishh/repos","events_url":"https://api.github.com/users/vishh/events{/privacy}","received_events_url":"https://api.github.com/users/vishh/received_events","type":"User","site_admin":false}],"milestone":null,"comments":57,"created_at":"2017-08-22T18:25:56Z","updated_at":"2018-05-07T23:40:57Z","closed_at":null,"author_association":"OWNER","body":"The effect of CPU throttling is non obvious to users and throws them off when they try out kubernetes. It also complicates CPU capacity planning for pods. Pods that are carefully placed in Guaranteed QoS class are typically of higher priority than other pods, and these pods are expected to get the best performance. On the other hand, Burstable pods may have to be limited in certain environments to avoid tail latencies for Guaranteed pods.\r\nI propose we avoid setting CPU limits on Guaranteed pods by default and continue to impose CPU limits on Burstable pods. We can provide a config option to ease the rollout in existing clusters. Users may need time to handle this transition, and some of them may choose to use the existing policies. \r\n\r\nClosely related is static CPU pinning policies in kubelet which will naturally limit Guaranteed pods to specific cores if they request integral number of cores. This feature may be combined with turning off CPU limiting for Guaranteed pods. That said, having an ability to configure each of these feature independently will still be helpful.\r\n\r\n@derekwaynecarr we have discussed this in the past, but AFAIK, an issue hasn't been filed thus far. @dchen1107 thoughts? \r\n\r\nIf there is a general agreement that this is the intended direction longer term, I can post a patch to make CPU limit policies configurable.\r\n\r\n@kubernetes/sig-node-feature-requests "},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/387239382","html_url":"https://github.com/kubernetes/kubernetes/issues/51135#issuecomment-387239382","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/51135","id":387239382,"user":{"login":"timoreimann","id":694826,"avatar_url":"https://avatars1.githubusercontent.com/u/694826?v=4","gravatar_id":"","url":"https://api.github.com/users/timoreimann","html_url":"https://github.com/timoreimann","followers_url":"https://api.github.com/users/timoreimann/followers","following_url":"https://api.github.com/users/timoreimann/following{/other_user}","gists_url":"https://api.github.com/users/timoreimann/gists{/gist_id}","starred_url":"https://api.github.com/users/timoreimann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/timoreimann/subscriptions","organizations_url":"https://api.github.com/users/timoreimann/orgs","repos_url":"https://api.github.com/users/timoreimann/repos","events_url":"https://api.github.com/users/timoreimann/events{/privacy}","received_events_url":"https://api.github.com/users/timoreimann/received_events","type":"User","site_admin":false},"created_at":"2018-05-07T23:40:57Z","updated_at":"2018-05-07T23:40:57Z","author_association":"CONTRIBUTOR","body":"@vishh thanks for summarizing our discussion at KubeCon EU. I'd like to reiterate one angle that reflects our use case: One goal for us is to protect the cluster from applications that are buggy to the extent that they request CPUs far beyond what could be considered occasional, short-termed spikes.\r\n\r\nOur (well-behaving) applications generally run fine resource-wise, despite the fact that we overcommit the cluster in a rather homogeneous way by means of specifying CPU requests at 500 mCPUs for the majority of applications. This works fine for us since we tend to keep enough slack CPU around on each node to account for bursts. Just the other day, however, we had a misbehaving application (configured without limits) that would generate a load of as much as 40 on an 8 core machine over a longer period of time, forcing every pod on that node into fair-sharing using weights that we haven't optimized for this situation. With limits, we can constrain the same application to something like 1.5 CPUs and be reasonably certain that it won't be able to hog the machine's CPU resources as easily.\r\n\r\nGiven such circumstances and a setup like ours, would a limitless environment along with a buffer DaemonSet really provide similar guarantees? Having an application going overboard as in our example, I understand that all applications would be forced into fair-sharing anyway, getting us back to the undesirable state where CPU requests must reflect actual application behavior relatively well (ideally by being Guaranteed pods) to provide a similar level of protection.\r\n\r\nWith regards to auto-scaling, I personally wouldn't consider that to be a good mitigation strategy in this case for several reasons (caveat: we don't use auto-scaling yet, so my concerns are more of a theoretical nature): First, there is likely some non-trivial latency involved to bring up more pods and especially more machines until the cluster could offer sufficient resources again; second, I assume it would also auto-scale misbehaving applications unless special care was taken to identify and exclude such apps; and third, it's a pretty big gun for a problem that  limits could presumably solve much easier for us (assuming a proper cycle time could be achieved).\r\n\r\nLet me know if I missed something. Happy to hear your thoughts."}}
{"actor":{"display_login":"timoreimann"}}