<p>We've experienced 4 <code>AUTO_REPAIR_NODES</code> events(revealed by the command <code>gcloud container operations list</code>) on our GKE cluster during the past 1 month. The consequence of node-auto-repair is that the node gets recreated and gets attached a new external IP, and the new external IP, which was not whitelisted by third-party services, eventually caused failure of services running on that the new node.</p>

<p>I noticed that we have "<strong>Automatic node repair</strong>" enabled in our Kubernetes cluster and felt tempted to disable that, but before I do that, I need to know more about the situation. </p>

<p>My questions are:</p>

<ol>
<li>What are some common causes that makes a node unhealthy in the first place? I'm aware of this article <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/node-auto-repair#node_repair_process" rel="nofollow noreferrer">https://cloud.google.com/kubernetes-engine/docs/how-to/node-auto-repair#node_repair_process</a> which says, "a node reports a <strong>NotReady</strong> status on consecutive checks over the given time threshold" would trigger auto repair. But what could cause a node to become <strong>NotReady</strong>?</li>
<li>I'm also aware of this article <a href="https://kubernetes.io/docs/concepts/architecture/nodes/#node-status" rel="nofollow noreferrer">https://kubernetes.io/docs/concepts/architecture/nodes/#node-status</a> which mentions the full list of node status: {OutOfDisk, Ready, MemoryPressure, PIDPressure, DiskPressure, NetworkUnavailable, ConfigOK}. I wonder, if any of {OutOfDisk, MemoryPressure, PIDPressure, DiskPressure, NetworkUnavailable} becomes true for a node, would that node becomes NotReady?</li>
<li>What negative consequences could I get after I disable "Automatic node repair" in the cluster? <strong>I'm basically wondering whether we could end up in a worse situation than auto-repaired nodes and newly-attached-not-whitelisted IP</strong>. Once "Automatic node repair" is disabled, then for the pods that are running on an Unhealthy node that would've been auto-repaired, would Kubernetes create new pods on other nodes?</li>
</ol>

<p>The confusion lies here in that there are 'Ready' and 'NotReady' states that are shown when you run <code>kubectl get nodes</code> which are reported by the kube-apiserver. But these are independent and unclear from the docs how they relate to the kubelet states described <a href="https://kubernetes.io/docs/concepts/architecture/nodes/#node-status" rel="nofollow noreferrer">here</a>
You can also see the kubelet states (in events) when you run <code>kubectl describe nodes</code> </p>

<p>To answer some parts of the questions:</p>

<ol>
<li><p>As reported by the kube-apiserver</p>

<ul>
<li>Kubelet down</li>
<li>docker or containerd or crio down (depending on the shim you are using)</li>
<li>kubelet states - unclear.</li>
</ul></li>
<li><p>For these, the kubelet will start evicting or not scheduling pods except for Ready (<a href="https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/" rel="nofollow noreferrer">https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/</a>). Unclear from the docs how these get reported from the kubeapi-server.</p></li>
<li><ul>
<li>You could have nodes on your cluster not being used and you'd be paying for that usage.</li>
<li>Yes, k8s will reschedule the pods after a certain readiness probes fail (configurable). If the kubelet is down or the node down k8s will think the pods are down.</li>
<li>Assuming your nodes go down, you could end up with less capacity than what you need to schedule your workloads to k8s would not be able to schedule them anyway.</li>
</ul></li>
</ol>

<p>Hope it helps!</p>
