<p>We have a Kubernetes cluster.</p>

<p>Now we want to expand that with GPU nodes (so that would be the only nodes in the Kubernetes cluster that have GPUs).</p>

<p>We'd like to avoid Kubernetes to schedule pods on those nodes unless they require GPUs. </p>

<p>Not all of our pipelines can use GPUs. The absolute majority are still CPU-heavy only. </p>

<p>The servers with GPUs could be very expensive (for example, Nvidia DGX could be as much as $150/k per server).</p>

<p>If we just add DGX nodes to Kubernetes cluster, then Kubernetes would schedule non-GPU workloads there too, which would be a waste of resources (e.g. other jobs that are getting scheduled later and do need GPUs, may have other non-GPU resources there exhausted there like CPU and memory, so they would have to wait for non-GPU jobs/containers to finish).</p>

<p>Is there is a way to customize GPU resource scheduling in Kubernetes so that it would only schedule pods on those expensive nodes if they require GPUs? If they don't, they may have to wait for availability of other non-GPU resources like CPU and memory on non-GPU servers... </p>

<p>Thanks.</p>

<p>You can use labels and label selectors for this.
<a href="https://kubernetes.io/docs/concepts/configuration/assign-pod-node/" rel="nofollow noreferrer">kubernates docs</a></p>

<p>Update: example </p>

<pre><code>apiVersion: v1
kind: Pod
metadata:
  name: with-gpu-antiAffinity
spec:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchExpressions:
            - key: resources
              operator: In
              values:
              - cpu-only
</code></pre>
