<p>I have a Job Object which shall use a Node Selector to only use Nodes, which have a GPU under the hood. I know to to set it (it gets converted from a string in a python program).</p>
<pre><code>    job = f&quot;&quot;&quot;
    apiVersion: batch/v1
    kind: Job
    ....
          nodeSelector:
            sma-gpu-size: {gpu_size}
    &quot;&quot;&quot;
</code></pre>
<p>Our ops team sets this selectors in the next few weeks, but currently when setting the node selector, the service is not able to start.</p>
<pre><code>2022-09-20T07:20:24Z [Warning] 0/35 nodes are available: 2 node(s) had taint {node-role.kubernetes.io/infra: }, that the pod didn't tolerate, 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 30 node(s) didn't match Pod's node affinity/selector.
</code></pre>
<p>Is it somehow possible to use these node_selectors only if they are available, something like this (pseudo yaml)?</p>
<pre><code>    job = f&quot;&quot;&quot;
    apiVersion: batch/v1
    kind: Job
    ....
          nodeSelector:
            if_available:
                sma-gpu-size: {gpu_size}
            else:
                Any
    &quot;&quot;&quot;
</code></pre>

Data Mastery
<p>It's not, but you can replace the <code>nodeSelector</code> with a <code>nodeAffinity</code> to achieve that.</p>
<pre><code>spec:
  [...]
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: sma-gpu-size
            operator: In
            values:
            - {gpu_size}
</code></pre>
<p>From <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/" rel="nofollow noreferrer">docs</a>:</p>
<blockquote>
<p>preferredDuringSchedulingIgnoredDuringExecution: The scheduler tries to find a node that meets the rule. If a matching node is not available, the scheduler still schedules the Pod.</p>
</blockquote>
<p>After the label has been added, you can switch to <code>requiredDuringSchedulingIgnoredDuringExecution</code>:</p>
<blockquote>
<p>requiredDuringSchedulingIgnoredDuringExecution: The scheduler can't schedule the Pod unless the rule is met. This functions like nodeSelector, but with a more expressive syntax.</p>
</blockquote>
<p>or back to <code>nodeSelector</code>.</p>
