<p>Our GKE Autopilot cluster was recently upgraded to version 1.21.6-gke.1503, which apparently causes the <code>cluster-autoscaler.kubernetes.io/safe-to-evict=false</code> annotation to be banned.</p>
<p>I totally get this for deployments, as Google doesn't want a deployment preventing scale-down, but for jobs I'd argue this annotation makes perfect sense in certain cases. We start complex jobs that start and monitor other jobs themselves, which makes it hard to make them restart-resistant given the sheer number of moving parts.</p>
<p><strong>Is there any way to make it as unlikely as possible for job pods to be restarted/moved around when using Autopilot?</strong> Prior to switching to Autopilot, we used to make sure our jobs filled a single node by requesting all of its available resources; combined with a Guaranteed QoS class, this made sure the only way for a pod to be evicted was if the node somehow failed, which almost never happened. Now all we seem to have left is the Guaranteed QoS class, but that doesn't prevent pods from being evicted.</p>

<p>At this point the only thing left is to ask to bring back this feature on <a href="https://issuetracker.google.com" rel="nofollow noreferrer">IssueTracker</a> - raise a new feature reqest and hope for the best.</p>
<p>Link to this thread also as it contains quite a lot of troubleshooting and may be useful.</p>
