<p>I am experiencing a lot of CPU throttling (see nginx graph below, other pods often 25% to 50%) in my Kubernetes cluster (k8s v1.18.12, running 4.9.0-11-amd64 #1 SMP Debian 4.9.189-3+deb9u2 (2019-11-11) x86_64 GNU/Linux).</p>
<p>Due to backports, I do not know whether my cluster contains the Linux kernel bug described in <a href="https://lkml.org/lkml/2019/5/17/581" rel="nofollow noreferrer">https://lkml.org/lkml/2019/5/17/581</a>. How can I find out? Is there a simple way to check or measure?</p>
<p>If I have the bug, what is the best approach to get the fix? Or should I mitigate otherwise, e.g. not use CFS quota (<code>--cpu-cfs-quota=false</code> or no CPU limits) or reduce <code>cfs_period_us</code> and <code>cfs_quota_us</code>?</p>
<p>CPU Throttling Percentage for nginx (scaling horizontally around 15:00 and removing CPU limits around 19:30): <a href="https://i.stack.imgur.com/M1oi3.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/M1oi3.png" alt="enter image description here" /></a></p>

<p>Recently, I'm working on debuging the cpu throttling issue, with the following 5 tests, I've tested out the bug in kernel (Linux version 4.18.0-041800rc4-generic)</p>
<p>This test case is intended to hit 100% throttling for the test 5000ms / 100 ms periods = 50 periods. A kernel without this bug should be able to have a CPU usage stats about 500ms.</p>
<p>Maybe you can try these tests to check whether your kernel will be throttlled.</p>
<p>[Multi Thread Test 1]</p>
<pre><code>./runfibtest 1; ./runfibtest

From &lt;https://github.com/indeedeng/fibtest&gt; 
</code></pre>
<p>[Result]</p>
<pre><code>Throttled

./runfibtest 1
Iterations Completed(M): 465 
Throttled for: 52 
CPU Usage (msecs) = 508

./runfibtest 8
Iterations Completed(M): 283 
Throttled for: 52 
CPU Usage (msecs) = 327
</code></pre>
<p>[Multi Thread Test 2]</p>
<pre><code>docker run -it --rm --cpu-quota 10000 --cpu-period 100000 hipeteryang/fibtest:latest /bin/sh -c &quot;runfibtest 8 &amp;&amp; cat /sys/fs/cgroup/cpu,cpuacct/cpu.stat &amp;&amp; cat /sys/fs/cgroup/cpu,cpuacct/cpuacct.usage &amp;&amp; cat /sys/fs/cgroup/cpu,cpuacct/cpuacct.usage_percpu&quot;
</code></pre>
<p>[Result]</p>
<pre><code>Throttled

Iterations Completed(M): 192 
Throttled for: 53 
CPU Usage (msecs) = 227
nr_periods 58
nr_throttled 56
throttled_time 10136239267
267434463
209397643 2871651 8044402 4226146 5891926 5532789 27939741 4104364
</code></pre>
<p>[Multi Thread Test 3]</p>
<pre><code>docker run -it --rm --cpu-quota 10000 --cpu-period 100000 hipeteryang/stress-ng:cpu-delay /bin/sh -c &quot;stress-ng --taskset 0 --cpu 1 --timeout 5s &amp; stress-ng  --taskset 1-7 --cpu 7 --cpu-load-slice -1 --cpu-delay 10 --cpu-method fibonacci  --timeout 5s &amp;&amp; cat /sys/fs/cgroup/cpu,cpuacct/cpu.stat &amp;&amp; cat /sys/fs/cgroup/cpu,cpuacct/cpuacct.usage &amp;&amp; cat /sys/fs/cgroup/cpu,cpuacct/cpuacct.usage_percpu&quot;
</code></pre>
<p>Result</p>
<pre><code>Throttled

nr_periods 56
nr_throttled 53
throttled_time 7893876370
379589091
330166879 3073914 6581265 724144 706787 5605273 29455102 3849694
</code></pre>
<p>For the following kubernetes test, we can use &quot;kubectl logs pod-name&quot; to get the result once the job is done</p>
<p>[Multi Thread Test 4]</p>
<pre><code>apiVersion: batch/v1
kind: Job
metadata:
  name: fibtest
  namespace: default
spec:
  template:
    spec:
      containers:
      - name: fibtest
        image: hipeteryang/fibtest
        command: [&quot;/bin/bash&quot;, &quot;-c&quot;, &quot;runfibtest 8 &amp;&amp; cat /sys/fs/cgroup/cpu,cpuacct/cpu.stat &amp;&amp; cat /sys/fs/cgroup/cpu,cpuacct/cpuacct.usage &amp;&amp; cat /sys/fs/cgroup/cpu,cpuacct/cpuacct.usage_percpu&quot;]
        resources:
          requests:
            cpu: &quot;50m&quot;
          limits:
            cpu: &quot;100m&quot;
      restartPolicy: Never
</code></pre>
<p>Result</p>
<pre><code>Throttled

Iterations Completed(M): 195 
Throttled for: 52 
CPU Usage (msecs) = 230
nr_periods 56
nr_throttled 54
throttled_time 9667667360
255738571
213621103 4038989 2814638 15869649 4435168 5459186 4549675 5437010
</code></pre>
<p>[Multi Thread Test 5]</p>
<pre><code>apiVersion: batch/v1
kind: Job
metadata:
  name: stress-ng-test
  namespace: default
spec:
  template:
    spec:
      containers:
      - name: stress-ng-test
        image: hipeteryang/stress-ng:cpu-delay
        command: [&quot;/bin/bash&quot;, &quot;-c&quot;, &quot;stress-ng --taskset 0 --cpu 1 --timeout 5s &amp; stress-ng  --taskset 1-7 --cpu 7 --cpu-load-slice -1 --cpu-delay 10 --cpu-method fibonacci  --timeout 5s &amp;&amp; cat /sys/fs/cgroup/cpu,cpuacct/cpu.stat &amp;&amp; cat /sys/fs/cgroup/cpu,cpuacct/cpuacct.usage &amp;&amp; cat /sys/fs/cgroup/cpu,cpuacct/cpuacct.usage_percpu
&quot;]
        resources:
          requests:
            cpu: &quot;50m&quot;
          limits:
            cpu: &quot;100m&quot;
      restartPolicy: Never
</code></pre>
<p>Result</p>
<pre><code>Throttled

nr_periods 53
nr_throttled 50
throttled_time 6827221694
417331622
381601924 1267814 8332150 3933171 13654620 184120 6376208 2623172
</code></pre>
<p>Feel free to leave any comment, I’ll reply as soon as possible.</p>
