<p>I'm experiencing a <strong>strange issue</strong> when using <strong>CPU Requests/Limits in Kubernetes</strong>. Prior to setting any CPU Requests/Limits at all, all my services performed very well. I recently started placing some Resource Quotas to avoid future resource starvation. These values were set based in the actual usage of those services, but to my surprise, after those were added, some <strong>services started to increase their response time drastically</strong>. My first guess was that I might placed wrong Requests/Limits, but looking at the metrics revealed that in fact <strong>none of the services facing this issue were near those values</strong>. In fact, some of them were closer to the Requests than the Limits. </p>

<p>Then I started looking at CPU throttling metrics and found that <strong>all my pods are being throttled</strong>. I then increased the limits for one of the services to 1000m (from 250m) and I saw less throttling in that pod, but I don't understand why I should set that higher limit if the pod wasn't reaching its old limit (250m).</p>

<p>So my question is: <strong>If I'm not reaching the CPU limits, why are my pods throttling? Why is my response time increasing if the pods are not using their full capacity?</strong></p>

<p>Here there are some screenshots of my metrics <strong>(CPU Request: 50m, CPU Limit: 250m)</strong>:</p>

<p><strong>CPU Usage (here we can see the CPU of this pod never reached its limit of 250m):</strong>
<a href="https://i.stack.imgur.com/3W7ig.png" rel="noreferrer"><img src="https://i.stack.imgur.com/3W7ig.png" alt="CPU Usage"></a></p>

<p><strong>CPU Throttling:</strong>
<a href="https://i.stack.imgur.com/XaPIf.png" rel="noreferrer"><img src="https://i.stack.imgur.com/XaPIf.png" alt="CPU Throttling"></a></p>

<p><strong>After setting limits to this pod to <em>1000m</em>, we can observe less throttling</strong>
<a href="https://i.stack.imgur.com/XfQxU.png" rel="noreferrer"><img src="https://i.stack.imgur.com/XfQxU.png" alt="Comparation"></a></p>

<p><strong>kubectl top</strong></p>

<p><a href="https://i.stack.imgur.com/mZWoi.png" rel="noreferrer"><img src="https://i.stack.imgur.com/mZWoi.png" alt="Top"></a></p>

<p>P.S: Before setting these Requests/Limits there wasn't throttling at all (as expected)</p>

<p>P.S 2: None of my nodes are facing high usage. In fact, none of them are using more than 50% of CPU at any time.</p>

<p>Thanks in advance!</p>

<p>If you see the <a href="https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#how-pods-with-resource-limits-are-run" rel="noreferrer">documentation</a> you see when you issue a <code>Request</code> for CPUs it actually uses the <a href="https://docs.docker.com/engine/reference/run/#cpu-share-constraint" rel="noreferrer"><code>--cpu-shares</code></a> option in Docker which actually uses the cpu.shares attribute for the <a href="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/resource_management_guide/sec-cpu_and_memory-use_case" rel="noreferrer">cpu,cpuacct cgroup</a> on Linux. So a value of <code>50m</code> is about <code>--cpu-shares=51</code> based on the maximum being <code>1024</code>.  <code>1024</code> represents 100% of the shares, so <code>51</code> would be 4-5% of the share.  That's pretty low, to begin with.  But the important factor here is that this relative to how many pods/container you have on your system and what cpu-shares those have (are they using the default).</p>
<p>So let's say that on your node you have another pod/container with 1024 shares which is the default and you have this pod/container with 4-5 shares. Then this container will get about get about 0.5% CPU, while the other  pod/container will
get about 99.5% of the CPU (if it has no limits). So again it all depends on how many pods/container you have on the node and what their shares are.</p>
<p>Also, not very well documented in the <a href="https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#how-pods-with-resource-limits-are-run" rel="noreferrer">Kubernetes docs</a>, but if you use <code>Limit</code> on a pod it's basically using two flags in Docker: <a href="https://docs.docker.com/engine/reference/run/#cpu-period-constraint" rel="noreferrer"><code>--cpu-period and --cpu--quota</code></a> which actually uses the cpu.cfs_period_us and the cpu.cfs_quota_us attributes for the <a href="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/resource_management_guide/sec-cpu_and_memory-use_case" rel="noreferrer">cpu,cpuacct cgroup</a> on Linux. This was introduced to the fact that cpu.shares didn't provide a limit so you'd spill over cases where containers would grab most of the CPU.</p>
<p>So, as far as this limit is concerned you will never hit it if you have other containers on the same node that don't have limits (or higher limits) but have a higher cpu.shares because they will end up optimizing and picking idle CPU. This could be what you are seeing, but again depends on your specific case.</p>
<p>A longer explanation for all of the above <a href="https://medium.com/@betz.mark/understanding-resource-limits-in-kubernetes-cpu-time-9eff74d3161b" rel="noreferrer">here</a>.</p>
