<p>I have a on-premise k8s GPU cluster with several computing nodes which have 8 GPUs respectively. Because we are on the way of migration to k8s, and also there are some remained project to use GPUs, I have to remain some GPUs which do not allowed by k8s to use. Furthermore, some projects require the use of contiguous GPUs(I mean, using a physically adjacent PCIE bus), so it is necessary to leave those GPUs specific.</p>
<p>So the question is, is it possible to force k8s worker to use specific GPU only? For example, I would like to configure my k8s worker to allocate GPUs from 0 to 4 and GPUs from 4 to 7 cannot be allocated.</p>
<p>Thanks!</p>

frstck
<p>if you want your jobs to never use one of the gpu nodes, you can taint this nodes with noschedule taint
for example:</p>
<pre><code>kubectl taint nodes aks-gpuv100small.. no=k8s:NoSchedule  
</code></pre>
<p>if you want your jobs to run only on specific gpu nodes you can add labels to this nodes and nodeSelectors to your jobs</p>
<pre><code> kubectl label nodes aks-gpuv100small.. k8s=gpu
 nodeSelector:
   k8s: gpu
</code></pre>
<p>if you want inside a gpu node to only expose some of the devices to the job you can add the following environment variable to your job</p>
<pre><code>CUDA_VISIBLE_DEVICES=0,1,2,3
</code></pre>
