<p>I've been having a daily issue with my Kubernetes cluster (running on 1.18) where one of the nodes will go over 100% CPU utilisation, and Kubernetes will fail to connect external visitors to my pods. (A website outage, basically)</p>
<p>The strange thing is the pods are always sitting at a comfortable 30% (or lower!) CPU. So the application itself seems okay.</p>
<p>When I <code>describe</code> the node in question, I see mention of a <code>node-problem-detector</code> timeout.</p>
<pre><code>Events:
  Type     Reason                  Age                      From                                     Message
  ----     ------                  ---                      ----                                     -------
  Normal   NodeNotSchedulable      10m                      kubelet                                  Node nodepoo1-vmss000007 status is now: NodeNotSchedulable
  Warning  KubeletIsDown           9m44s (x63 over 5h21m)   kubelet-custom-plugin-monitor            Timeout when running plugin &quot;/etc/node-problem-detector.d/plugin/check_kubelet.s&quot;
  Warning  ContainerRuntimeIsDown  9m41s (x238 over 5h25m)  container-runtime-custom-plugin-monitor  Timeout when running plugin &quot;/etc/node-problem-detector.d/plugin/check_runtime.s&quot;
</code></pre>
<p>My current approach has been to run three nodes on my nodepool, and effectively babysit Kubernetes by cordoning the troublesome node and moving all the pods onto one of the other nodes during the monitoring outage. After 15 minutes once things are back to normal, I will uncordon the effected node and start the cycle again.</p>
<p>I was particularly unlucky this weekend where I had three CPU spikes within 24 hours.</p>
<p><a href="https://i.stack.imgur.com/vjNeO.jpg" rel="noreferrer"><img src="https://i.stack.imgur.com/vjNeO.jpg" alt="CPU chart showing spike from one of three nodes" /></a></p>
<p>How can I go about fixing this issue? I can't seem to find any information on the <code>Timeout when running plugin &quot;/etc/node-problem-detector.d/plugin/check_kubelet.s&quot;</code> problem I'm seeing.</p>

<p>You could try to open an <code>ssh</code> connection to the node and then check which process(es) consume CPU using <code>top</code>. If this process runs in a pod and your have <code>crictl</code> installed on the node, you can use <a href="https://github.com/k8s-school/pid2pod" rel="nofollow noreferrer">https://github.com/k8s-school/pid2pod</a> to retrieve the pod which is running the process.</p>
