<p>I have a service deployed in Kubernetes and I am trying to optimize the <a href="https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#resource-requests-and-limits-of-pod-and-container" rel="nofollow noreferrer">requested cpu resources</a>.</p>

<p>For now, I have deployed 10 instances and set <code>spec.containers[].resources.limits.cpu</code> to <code>0.1</code>, based on the "average" use. However, it became obvious that this average is rather useless in practice because under constant load, the load increases significantly (to 0.3-0.4 as far as I can tell).</p>

<p>What happens consequently, when multiple instances are deployed on the same node, is that this node is heavily overloaded; pods are no longer responsive, are killed and restarted etc.</p>

<p>What is the best practice to find a good value? My current best guess is to increase the requested cpu to 0.3 or 0.4; I'm looking at Grafana visualizations and see that the pods on the heavily loaded node(s) converge there under continuous load.
However, how can I know if they would use more load if they could before becoming unresponsive as the node is overloaded?</p>

<p>I'm actually trying to understand how to approach this in general. I would expect an "ideal" service (presuming it is CPU-focused) to use close to <code>0.0</code> when there is no load, and close to 1.0 when requests are constantly coming in. With that assumption, should I set the <code>cpu.requests</code> to <code>1.0</code>, taking a perspective where actual constant usage is assumed?</p>

<p>I have read some <a href="https://cloud.google.com/blog/products/gcp/kubernetes-best-practices-resource-requests-and-limits" rel="nofollow noreferrer">Kubernetes best practice guides</a>, but none of them seem to address how to set the actual value for cpu requests in practice in more depth than "find an average".</p>

Carsten
<p>In a nutshell: <strong>Main goal is to understand how much traffic a pod can handle and how much resource it consumes to do so.</strong></p>
<p>CPU limits are hard to understand and can be harmful, you might want
to avoid them, see <a href="https://v1-16.docs.kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/" rel="nofollow noreferrer">static policy documentation</a> and <a href="https://github.com/kubernetes/kubernetes/issues/51135" rel="nofollow noreferrer">relevant
github issue</a>.</p>
<p>To dimension your CPU requests you will want to understand first how much a pod can consume during high load. In order to do this you can :</p>
<ul>
<li>disable all kind of autoscaling (HPA, vertical pod autoscaler, ...)</li>
<li>set the number of replicas to one</li>
<li>lift the CPU limits</li>
<li>request the highest amount of CPU you can on a node (3.2 usually on 4cpu nodes)</li>
<li>send as much traffic as you can on the application (you can achieve simple Load Tests scenarios with <a href="https://locust.io/" rel="nofollow noreferrer">locust</a> for example)</li>
</ul>
<p>You will eventually end up with a ratio <code>clients-or-requests-per-sec/cpu-consumed</code>. You can suppose the relation is linear <em>(this might not be true if your workload complexity is O(n^2) with n the number of clients connected, but this is not the nominal case)</em>.</p>
<p>You can then choose the pod resource requests based on the ratio you measured. For example if you consume 1.2 cpu for 1000 requests per second you know that you can give each pod 1 cpu and it will handle up to 800 requests per second.</p>
<p>Once you know how much a pod can consume under its maximal load, you can start setting up cpu-based autoscaling, 70% is a good first target that can be refined if you encounter issues like latency or pods not autoscaling fast enough. This will avoid your nodes to run out of cpu if the load increases.</p>
<p>There are a few gotchas, for example <strong>single-threaded applications are not able to consume more than a cpu</strong>. Thus if you give it 1.5 cpu it will run out of cpu but you won't be able to visualize it from metrics as you'll believe it still can consume 0.5 cpu.</p>
