<p>I have some nodes with different amount of gpu cards. I want to be able to assign a specific pod to a certain gpu card, something like a reservation when I am assigning a task to it.</p>

<p>I have already seen, that you can label a node like that:</p>

<pre><code>kubectl label node nodaA project-type=ProjectA
</code></pre>

<p>But this is just labeling a node, which a pod will be assigned to it. But I want to be able to be more specific.</p>

<p>Let's say I have a node called <strong>NodeA</strong>, which has <strong>3 GPU cards</strong> from <strong>Nvidia</strong>. The first card is reserved for a specific task called "<strong><em>CertainOne</em></strong>", the other two cards are just there available for other pods. </p>

<p>When a user submits pods which is <strong>not</strong> the "<strong><em>CertainOne</em></strong>", the pods should be assigned to these 3 cards, no matter what label. But when another user submits a pod with the label "<strong><em>CertainOne</em></strong>", then  the reserved card should always be used (the other running pod which could be using at that moment should be dropped and the pod with the label CertainOne should use it as a priority)</p>

<p>But I wasn't able to find a solution for that. The whole idea is, that I can change the reservation card at anytime, for example to another card, but also to change the range, let's say not only the first one, but also the second one, so that pods with the label "<strong><em>CertainOne</em></strong>" should be assigned to card 1 or card 2 in this second scenario.</p>

<p>Is this even possible? Or is there another similar solution to that?</p>

TheHeroOfTime
<blockquote>
  <p>But this is just labeling a node, which a pod will be assigned to it.
  But I want to be able to be more specific.</p>
  
  <p>But I wasn't able to find a solution for that. </p>
  
  <p>Is this even possible? Or is there another similar solution to that?</p>
</blockquote>

<p>Unfortunately, as per my experience, I have bad news for you cause I have never seen solutions like what you want.
The very standard way is to use <a href="https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#node-affinity-beta-feature" rel="nofollow noreferrer"><code>NodeAffinity</code></a> in spike with appropriate labels and selectors. But this guarantee you scheduling pods on chosen nodes only and this doesnt fit you requirements :(</p>

<pre><code>$ kubectl label nodes nodeA k8s-node=gpunode

apiVersion: v1
kind: Pod
metadata:
  name: runongpunode
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: k8s-node
            operator: In
            values:
            - gpunode
</code></pre>
