<p>The problem:</p>
<p>We have nodes in kubernetes, which will occasionally become tainted with an <code>effect</code> of <code>PreferNoSchedule</code>. When this happens, we would like our pods to <em>completely</em> avoid scheduling on these nodes (in other words, act like the taint's <code>effect</code> were actually <code>NoSchedule</code>). The taints aren't applied by us - we're using GKE and it's an automated thing on their end, so we're stuck with the <code>PreferNoSchedule</code> behaviour.</p>
<p>What we <em>can</em> control is the spec of our pods. I'm hoping this might be possible using a <code>nodeAffinity</code> on them, however the documentation on this is fairly sparse: see e.g. <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinity" rel="nofollow noreferrer">here</a>. All the examples I can find refer to an affinity by labels, so I'm not sure if a taint is even visible/accessible by this logic.</p>
<p>Effectively, in my pod spec I want to write something like:</p>
<pre><code>spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/taints *I've invented this, it doesn't work
            operator: NotIn
            values:
            - DeletionCandidateOfClusterAutoscaler
</code></pre>
<p>where <code>DeletionCandidateOfClusterAutoscaler</code> is the taint that we see applied. Is there a way to make this work?</p>
<p>The other approach we've thought about is a cronjob which looks for the <code>PreferNoSchedule</code> taint and adds our own <code>NoSchedule</code> taint on top... but that feels a little gross!</p>
<p>Any neat suggestions / workarounds would be appreciated!</p>
<p>The long version (if you're interested):</p>
<p>The taint gets applied by the autoscaler to say the node is going to go away in 30 minutes or so. This issue describes in some more detail, from people having the same trouble: <a href="https://github.com/kubernetes/autoscaler/issues/2967" rel="nofollow noreferrer">https://github.com/kubernetes/autoscaler/issues/2967</a></p>

<p>I have tested your same scenario and indeed, GKE reconciles the current node configuration at the moment the autoscaler starts to spin up. This is in order to ensure no downtime in case of a lack of resources in the node which the pods/workloads can be scheduled. I believe there is no way to set the hard NoSchedule taint cleanly.</p>
<p>So, the critical information to keep in mind when using the autoscaler is:</p>
<ul>
<li><p>Pods will not be scheduled to the soft-tainted nodepool if there are resources available in the regular one.</p>
</li>
<li><p>If not enough resources are available in the regular one, they will be scheduled to the soft-tainted nodepool.</p>
</li>
<li><p>If there aren’t enough resources in the nodepools, the nodepool with the smallest nodetype will be autoscaled no matter the taints.</p>
</li>
</ul>
<p>As you mentioned a dirty workaround will be to:</p>
<p>A.- Create a cron or daemon to set the  NoSchedule and overwrite the soft one set by the autoscaler.</p>
<p>B.- A.- Ensure the resources habitability, maybe by setting perfect resource limits, and requests.</p>
