<p>I have a Jenkins deployment pipeline which involves kubernetes plugin. Using kubernetes plugin I create a slave pod for building a node application using <strong>yarn</strong>. The requests and limits for CPU and Memory are set.</p>

<p>When the Jenkins master schedules the slave, sometimes (as I haven’t seen a pattern, as of now), the pod makes the entire node unreachable and changes the status of node to be Unknown. On careful inspection in Grafana, the CPU and Memory Resources seem to be well within the range with no visible spike. The only spike that occurs is with the Disk I/O, which peaks to ~ 4 MiB.</p>

<p>I am not sure if that is the reason for the node unable to address itself as a cluster member. I would be needing help in a few things here:</p>

<p>a) How to diagnose in depth the reasons for node leaving the cluster.</p>

<p>b) If, the reason is Disk IOPS, is there any default requests, limits for IOPS at Kubernetes level?</p>

<p>PS: I am using EBS (gp2)</p>

Arpit Goyal
<p>As per the <a href="https://kubernetes.io/docs/concepts/architecture/nodes/#condition" rel="nofollow noreferrer">docs</a>, for the node to be 'Ready':</p>

<blockquote>
  <p>True if the node is healthy and ready to accept pods, False if the node is not healthy and is not accepting pods, and Unknown if the node controller has not heard from the node in the last node-monitor-grace-period (default is 40 seconds)</p>
</blockquote>

<p>If would seem that when you run your workloads your kube-apiserver doesn't hear from your node (kubelet) in 40 seconds. There could be multiple reasons, some things that you can try:</p>

<ul>
<li><p>To see the 'Events' in your node run:</p>

<pre><code>$ kubectl describe node &lt;node-name&gt;
</code></pre></li>
<li><p>To see if you see anything unusual on your kube-apiserver. On your active master run:</p>

<pre><code>$ docker logs &lt;container-id-of-kube-apiserver&gt;
</code></pre></li>
<li><p>To see if you see anything unusual on your kube-controller-manager when your node goes into 'Unknown' state. On your active master run:</p>

<pre><code>$ docker logs &lt;container-id-of-kube-controller-manager&gt;
</code></pre></li>
<li><p>Increase the <code>--node-monitor-grace-period</code> option in your kube-controller-manager. You can add it to the command line in the <code>/etc/kubernetes/manifests/kube-controller-manager.yaml</code> and restart the <code>kube-controller-manager</code> container.</p></li>
<li><p>When the node is in the 'Unknown' state can you <code>ssh</code> into it and see if you can reach the <code>kubeapi-server</code>?  Both on <code>&lt;master-ip&gt;:6443</code> and also the <code>kubernetes.default.svc.cluster.local:443</code> endpoints.</p></li>
</ul>
