<p>I am using <a href="https://learn.microsoft.com/en-us/azure/aks/node-updates-kured" rel="nofollow noreferrer">Kured</a> to perform safe reboots of our nodes to upgrade the OS and kernel versions.
In my understanding, it works by cordoning and draining the node, and the pods are scheduled on a new node with the older version. After the reboot, the nodes are uncordoned and back to the ready state and the temporary worker nodes get deleted.</p>
<p>It was perfectly fine until yesterday when one of the nodes failed to upgrade to the latest kernel version. It was on 5.4.0-1058-azure last week after a successful upgrade and it should be on 5.4.0-1059-azure yesterday after the latest patch, but it is using the old version 5.4.0-1047-azure (which I think is the version of the temporary node that got created).</p>
<p>Upon checking the log analytics on azure, it says that it failed to scale down.</p>
<p>Reason: ScaleDownFailed</p>
<p>Message: failed to drain the node, aborting ScaleDown</p>
<p><a href="https://i.stack.imgur.com/NTw84.png" rel="nofollow noreferrer">Error message</a></p>
<p>Any idea on why this is happening?</p>

themochishifter
<p>Firstly, there is a little misunderstanding of the OS and Kernel patching process.</p>
<blockquote>
<p>In my understanding, it works by cordoning and draining the node, and the pods are scheduled on a new node with the older version.</p>
</blockquote>
<p>The new node that is/are added should come with the latest <a href="https://learn.microsoft.com/en-us/azure/aks/node-image-upgrade#check-if-your-node-pool-is-on-the-latest-node-image" rel="nofollow noreferrer">node image version</a> with latest security patches (which <em>usually</em> does not fall back to an older kernel version) available for the node pool. You can check out the AKS node image releases <a href="https://github.com/Azure/AKS/tree/2021-09-16/vhd-notes" rel="nofollow noreferrer">here</a>. <a href="https://learn.microsoft.com/en-us/azure/aks/node-updates-kured#node-upgrades" rel="nofollow noreferrer">Reference</a></p>
<p>However, it is <em><strong>not</strong></em> necessary that the pod(s) evicted by the drain operation from the node that is being rebooted at any point during the process has to land on the <em>surge node</em>. Evicted pod(S) might very well be scheduled on an existing node should the node fit the bill for scheduling these pods.</p>
<p>For every Pod that the scheduler discovers, the scheduler becomes responsible for finding the <strong>best Node for that Pod</strong> to run on. The scheduler reaches this placement decision taking into account the scheduling principles described <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/kube-scheduler/#scheduling" rel="nofollow noreferrer">here</a>.</p>
<p>The documentation, at the time of writing, might be a little misleading on this.</p>
<hr />
<p>About the error:</p>
<blockquote>
<p>Reason: ScaleDownFailed<br>
Message: failed to drain the node, aborting ScaleDown</p>
</blockquote>
<p>This might happen due to a number of reasons. Common ones might be:</p>
<ul>
<li><p>The scheduler could not find a suitable node to place evicted pods and the node pool could not scale up due to insufficient compute quota available. [<a href="https://learn.microsoft.com/en-us/azure/aks/upgrade-cluster#before-you-begin" rel="nofollow noreferrer">Reference</a>]</p>
</li>
<li><p>The scheduler could not find a suitable node to place evicted pods and the cluster could not scale up due to insufficient IP addresses in the node pool's subnet. [<a href="https://learn.microsoft.com/en-us/azure/aks/configure-azure-cni#plan-ip-addressing-for-your-cluster" rel="nofollow noreferrer">Reference</a>]</p>
</li>
<li><p><code>PodDisruptionBudgets</code> (PDBs) did not allow for at least 1 pod replica to be moved at a time causing the drain/evict operation to fail. [<a href="https://learn.microsoft.com/en-us/azure/aks/upgrade-cluster#upgrade-an-aks-cluster" rel="nofollow noreferrer">Reference</a>]</p>
</li>
</ul>
<p>In general,</p>
<p>The <a href="https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/#eviction-api" rel="nofollow noreferrer">Eviction API</a> can respond in one of three ways:</p>
<ul>
<li>If the eviction is granted, then the Pod is deleted as if you sent a <code>DELETE</code> request to the Pod's URL and received back <code>200 OK</code>.</li>
<li>If the current state of affairs wouldn't allow an eviction by the rules set forth in the budget, you get back <code>429 Too Many Requests</code>. This is typically used for generic rate limiting of <em>any</em> requests, but here we mean that this request isn't allowed <em>right now</em> but it may be allowed later.</li>
<li>If there is some kind of misconfiguration; for example multiple PodDisruptionBudgets that refer the same Pod, you get a <code>500 Internal Server Error</code> response.</li>
</ul>
<p>For a given eviction request, there are two cases:</p>
<ul>
<li>There is no budget that matches this pod. In this case, the server always returns <code>200 OK</code>.</li>
<li>There is at least one budget. In this case, any of the three above responses may apply.</li>
</ul>
<p><strong>Stuck evictions</strong><br>
In some cases, an application may reach a broken state, one where unless you intervene the eviction API will never return anything other than 429 or 500.</p>
<p>For example: this can happen if ReplicaSet is creating Pods for your application but the replacement Pods do not become <code>Ready</code>. You can also see similar symptoms if the last Pod evicted has a very long termination grace period.</p>
<hr />
<p><strong>How to investigate further?</strong></p>
<ol>
<li><p>On the Azure Portal navigate to your AKS cluster</p>
</li>
<li><p>Go to <strong>Resource Health</strong> on the left hand menu as shown below and click on <strong>Diagnose and solve problems</strong></p>
<p><a href="https://i.stack.imgur.com/pjz9X.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/pjz9X.png" alt="enter image description here" /></a></p>
</li>
<li><p>You should see something like the following</p>
<p><a href="https://i.stack.imgur.com/lRjwr.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/lRjwr.png" alt="enter image description here" /></a></p>
</li>
<li><p>If you click on each of the options, you should see a number of checks loading. You can set the time frame of impact on the top right hand corner of the screen as shown below (Please press the <code>Enter</code> key after you have set the correct timeframe). You can click on the <code>More Info</code> link on the right hand side of each entry for detailed information and recommended action.</p>
<p><a href="https://i.stack.imgur.com/BngP2.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/BngP2.png" alt="enter image description here" /></a></p>
</li>
</ol>
<p><strong>How to mitigate the issue?</strong></p>
<p>Once you have identified the issue and followed the recommendations to fix the same, please perform an <code>az aks upgrade</code> on the AKS cluster to the same Kubernetes version it is currently running. This should initiate a reconcile operation wherever required under the hood.</p>
