<p>I use Google Cloud Platform for my project.<br>
Now, I have a cluster with 4 node pools:<br>
- "micro-pool": with minimal machines for managing the cluster<br>
- "cpu-pool": with cpu-only machines for processes that don't need a GPU<br>
- 2 "gpu-pools": two pools with machines that have GPUs attached.  </p>

<p>Now, what I need is for my CPU processes to never work on a GPU machine because they take so much time and doing that on a GPU machine is just costing money for nothing.<br>
I run my pods using the<br>
<code>kubectl run dc-1 --image={image-name} --replicas=1 --restart=Never --limits="nvidia.com/gpu=0,cpu=4000m,memory=2Gi" -- bash -c "command to execute"</code>    </p>

<p>Now, this works fine if there were no "GPU-machines" created from previous GPU runs. But if there was a very recent GPU run, this command will run on that instance because it has the minimum cpu and memory requirements. I thought the <code>--limits="nvidia.com/gpu=0</code> would do the trick but obviously it didn't.  </p>

<p>What should I do? </p>

Ahmedn1
<p>This is a good use case for <a href="https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/" rel="nofollow noreferrer">taints and tolerations</a>. You can taint the GPU nodes with NoSchedule. This will prevent pods (even system pods) that don't have a toleration for that taint from running on the GPU nodes</p>
<pre><code>kubectl taint nodes gpuNode1 nodetype=gpu:NoSchedule
</code></pre>
<p>Then, on pods you do want to run on these nodes, you can add a toleration for the taint:</p>
<pre><code>tolerations:
- key: &quot;nodetype&quot;
  operator: &quot;Equal&quot;
  value: &quot;gpu&quot;
  effect: &quot;NoSchedule&quot;
</code></pre>
<p>I'm not sure on GCP, but on Azure's AKS you can configure the taint when you create the cluster and the node pools.</p>
<p>Edit:</p>
<p>You will want to combine this with Harsh Manvar's sugestion of node selectors and/or affinity. Just because your pod can tolerate the taint, doesn't mean it will be scheduled on the GPU nodes for sure, it will just make sure other things are not.</p>
