<p>I have an old nodepool with machine type X, and I am migrating them workload to a new nodepool of machine type Y. Both nodepools are up, and I've disabled autoscaling on the old nodepool. I added a cordon to the old nodepool, then drained the nodes using the command provided by <a href="https://cloud.google.com/kubernetes-engine/docs/tutorials/migrating-node-pool" rel="nofollow noreferrer">the GKE doc</a>:</p>
<pre><code>for node in $(kubectl get nodes -l cloud.google.com/gke-nodepool=old-pool -o=name); do
  kubectl drain --force --ignore-daemonsets --delete-emptydir-data --grace-period=300 &quot;$node&quot;;
done
</code></pre>
<p>Those have a pod disruption budget (PDB) of Min 1 available, and the new nodepool has autoscaling setup with min 1 and max around 10.</p>
<p>My problem is that the drain does not complete, it gets stuck saying <code>Cannot evict pod as it would violate the pod's disruption budget.</code>. It makes sense that it doesn't evict the pod, but at the same time, should it trigger the creation of the same pod on the new nodepool?</p>
<p>As new pods are not getting created on the new nodepool to let the old node drain, I cannot migrate my workload (at least not without disruption, which I cannot have).</p>
<p>What am I missing here?</p>

David Gourde
<p>The log <code>Cannot evict pod as it would violate the pod's disruption budget</code> confirms that the root cause is due to the pod's disruption budget(PDB) configuration.</p>
<p>The main components of the PDB are <strong>minavailable</strong> and <strong>maxunavailable</strong>.</p>
<p><code>spec.minAvailable</code> : this is the total number of pods that must be available after the eviction, in the absence of the evicted pod.</p>
<p><code>spec.maxUnavailable</code> : this is the maximum number of pods that can go unavailable during an eviction.</p>
<p>According to the official <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-autoscaler#scheduling-and-disruption" rel="nofollow noreferrer">documentation</a>, an application's PodDisruptionBudget can also prevent autoscaling; if deleting nodes would cause the budget to be exceeded, the cluster does not scale down.</p>
<p>When scaling down, the cluster autoscaler respects scheduling and eviction rules set on Pods. These restrictions can prevent a node from being deleted by the autoscaler. A node's deletion could be prevented if it contains a Pod with any of these conditions:</p>
<blockquote>
<ul>
<li>The Pod's <strong>affinity</strong> or <strong>anti-affinity</strong> rules prevent rescheduling.</li>
<li>The Pod is not managed by a <strong>Controller</strong> such as a Deployment,    StatefulSet, Job or ReplicaSet.</li>
<li>The Pod has local storage and the GKE control plane version is lower    than 1.22. In GKE clusters with control plane version 1.22 or
later,    Pods with local storage no longer block scaling down.</li>
<li>The Pod has the <strong>&quot;cluster-autoscaler.kubernetes.io/safe-to-evict&quot;:    &quot;false&quot;</strong> annotation.</li>
<li>The node's deletion would exceed the configured <strong>PodDisruptionBudget</strong>    and the operation is unable to complete because of a deployment which    is having PDB .From the behavior
identified in the PDB you can <strong>change</strong>    the <strong>minAvailable</strong> or
<strong>maxUnavailable</strong> values of the <strong>PDB</strong>.</li>
</ul>
</blockquote>
<p>Can you check the above conditions which are preventing the node’s deletion in your deployment along with PDB.</p>
<p>Hope the mentioned information is useful to you.</p>
