<p>I'm following <a href="https://cloud.google.com/kubernetes-engine/docs/tutorials/migrating-node-pool" rel="nofollow noreferrer">this guide</a> in an attempt to upgrade a kubernetes cluster on GKE with no downtime. I've gotten all the old nodes cordoned and most of the pods have been evicted, but for a couple of the nodes, <code>kubectl drain</code> just keeps running and not evicting any more pods.</p>

<p><code>kubectl get pods --all-namespaces -o=wide</code> shows a handful of pods still running on the old pool, and when I run <code>kubectl drain --ignore-daemonsets --force</code> it prints a warning explaining why it's ignoring most of them; the only ones it doesn't mention are the pods I have running memcached, which were created via helm using <a href="https://github.com/kubernetes/charts/tree/master/stable/memcached" rel="nofollow noreferrer">this chart</a>.</p>

<p>We don't rely too heavily on memcached, so I could just go ahead and delete the old node pool at this point and accept the brief downtime for that one service. But I'd prefer to have a script to do this whole thing the right way, and I wouldn't know what to do at this point if these pods were doing something more important.</p>

<p>So, is this expected behavior somehow? Is there something about that helm chart that's making these pods refuse to be evicted? Is there another force/ignore sort of flag I need to pass to <code>kubectl drain</code>?</p>

<p>The helm chart you linked contains a PodDisruptionBudget (PDB). <code>kubectl drain</code> will not remove pods if it would violate a PDB (reference: <a href="https://kubernetes.io/docs/concepts/workloads/pods/disruptions/" rel="nofollow noreferrer">https://kubernetes.io/docs/concepts/workloads/pods/disruptions/</a>, "How Disruption Budgets Work" section mentions this). </p>

<p>If <code>minAvailable</code> on your PDB equals to number of replicas of your pod you will not be able to drain the node. Given that <a href="https://github.com/kubernetes/charts/blob/master/stable/memcached/values.yaml" rel="nofollow noreferrer">https://github.com/kubernetes/charts/blob/master/stable/memcached/values.yaml</a> has both set to 3, I would guess that's most likely the source of your problem. Just set your PDB <code>minAvailable</code> to one less than the desired number of replicas and it will be able to move your pods one-by-one.</p>
