<p>Recently we faced some issue in our AKS cluster that the nodes memory were simply got increased as the pods memory request was high ( request-2Gi, memory 2Gi) which increased the node count. So inorder to reduce the node counts we reduced the request memory to 256MI and limit to same value (2GB). After this we noticed some strange behaviour in our cluster.</p>
<ol>
<li>there is big difference in % of request and limits of our resource.<br />
More clearly the Limits values are showing 602% and 478 % of Actual, and much difference
between request %.
Is it normal or nor harm to keep this difference between request and limit ?</li>
</ol>
<blockquote>
<pre><code> Resource                       Requests      Limits
  --------                       --------      ------
  cpu                            1895m (99%)   11450m (602%)
  memory                         3971Mi (86%)  21830Mi (478%)
  ephemeral-storage              0 (0%)        0 (0%)
  hugepages-1Gi                  0 (0%)        0 (0%)
  hugepages-2Mi                  0 (0%)        0 (0%)
  attachable-volumes-azure-disk  0             0
</code></pre>
</blockquote>
<ol start="2">
<li>We noticed that our nodes consumption of memory is showing more than 100 %, which is a starnge
behaviour that how a node can consume more memory than actually it has.</li>
</ol>
<blockquote>
<pre><code>NAME                                CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
aks-nodepoolx-xxxxxxxx-vmss00000x   151m         7%     5318Mi          116%
</code></pre>
</blockquote>
<pre><code>NAME                                    READY   STATUS    RESTARTS   AGE
mymobile-mobile-xxxxx-ddvd6   2/2     Running   0          151m
myappsvc-xxxxxxxxxx-2t6gz     2/2     Running   0          5h3m
myappsvc-xxxxxxxxxx-4xnsh     0/2     Evicted   0          4h38m
myappsvc-xxxxxxxxxx-5b5mb     0/2     Evicted   0          4h28m
myappsvc-xxxxxxxxxx-5f52g     0/2     Evicted   0          4h19m
myappsvc-xxxxxxxxxx-5f8rz     0/2     Evicted   0          4h31m
myappsvc-xxxxxxxxxx-66lc9     0/2     Evicted   0          4h26m
myappsvc-xxxxxxxxxx-8cnfb     0/2     Evicted   0          4h27m
myappsvc-xxxxxxxxxx-b9f9h     0/2     Evicted   0          4h20m
myappsvc-xxxxxxxxxx-dfx9m     0/2     Evicted   0          4h30m
myappsvc-xxxxxxxxxx-fpwg9     0/2     Evicted   0          4h25m
myappsvc-xxxxxxxxxx-kclt8     0/2     Evicted   0          4h22m
myappsvc-xxxxxxxxxx-kzmxw     0/2     Evicted   0          4h33m
myappsvc-xxxxxxxxxx-lrrnr     2/2     Running   0          4h18m
myappsvc-xxxxxxxxxx-lx4bn     0/2     Evicted   0          4h32m
myappsvc-xxxxxxxxxx-nsc8t     0/2     Evicted   0          4h29m
myappsvc-xxxxxxxxxx-qmlrj     0/2     Evicted   0          4h24m
myappsvc-xxxxxxxxxx-qr75w     0/2     Evicted   0          4h27m
myappsvc-xxxxxxxxxx-tf8bn     0/2     Evicted   0          4h20m
myappsvc-xxxxxxxxxx-vfcdv     0/2     Evicted   0          4h23m
myappsvc-xxxxxxxxxx-vltgw     0/2     Evicted   0          4h31m
myappsvc-xxxxxxxxxx-xhqtb     0/2     Evicted   0          4h22m
</code></pre>

<p>Here are lots of talks about <strong>removing CPU limits</strong> on K8S</p>
<ul>
<li><a href="https://home.robusta.dev/blog/stop-using-cpu-limits/" rel="noreferrer">For the love of god, stop using CPU limits on Kubernetes </a></li>
<li><a href="https://community.ibm.com/community/user/aiops/blogs/dina-henderson/2022/06/29/kubernetes-cpu-throttling-the-silent-killer-of-res" rel="noreferrer">Kubernetes CPU Throttling: The Silent Killer of Response Time</a></li>
<li><a href="https://erickhun.com/posts/kubernetes-faster-services-no-cpu-limits/" rel="noreferrer">Kubernetes: Make your services faster by removing CPU limits</a></li>
<li><a href="https://medium.com/omio-engineering/cpu-limits-and-aggressive-throttling-in-kubernetes-c5b20bd8a718" rel="noreferrer">CPU limits and aggressive throttling in Kubernetes</a></li>
</ul>
<hr />
<p>Best practices for <strong>CPU limits and requests</strong> on Kubernetes</p>
<ul>
<li>Use CPU requests for everything and make sure they are accurate</li>
<li>Do <strong>NOT</strong> use CPU limits.</li>
</ul>
<p>Best practices for <strong>Memory limits and requests</strong> on Kubernetes</p>
<ul>
<li>Use memory limits and memory requests</li>
<li>Set memory limit= memory request</li>
</ul>
<hr />
<p>Checking the <strong>throttling rate of your pods</strong></p>
<p>Just login to the pod and run <code>cat /sys/fs/cgroup/cpu,cpuacct/kubepods/{PODID}/{CONTAINERID}/cpu.stat</code>.</p>
<ul>
<li><code>nr_periods</code> — Total schedule period</li>
<li><code>nr_throttled</code> — Total throttled period out of nr_periods</li>
<li><code>throttled_time</code> — Total throttled time in ns</li>
</ul>
