<p>I am trying to assign a cluster of pods to nodepools and I would like those nodepools to change based on the resources requested by the cluster pods. However, I'd like the pods to prefer the smaller of the nodepools (worker) and ignore the larger nodes (lgworker) (so, do not trigger a scale up).</p>
<pre class="lang-yaml prettyprint-override"><code>        extraPodConfig:
          tolerations:
            - key: toleration_label
              value: worker
              operator: Equal
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                - matchExpressions:
                  - key: a_node_selector_label
                    operator: In
                    values:
                      - worker
              preferredDuringSchedulingIgnoredDuringExecution:
                - weight: 100
                  preference:
                    matchExpressions:
                    - key: node_label
                      operator: In
                      values: 
                      - worker
                - weight: 90
                  preference:
                    matchExpressions:
                    - key: node_label
                      operator: In
                      values: 
                      - lgworker


</code></pre>
<p>The cluster pods default resource requests will fit on the smaller nodes easily and so I want to cause that to be used first. The larger nodepool should only be triggered when more resources than would fit on the smaller node be requested.</p>
<p>I have tried to weight the preferences, however the default cluster pods are being scheduled on to the larger nodepool.</p>
<p>Is there something I am missing that would help me properly assign pods to the smaller nodes over the larger nodes?</p>

Snow24
<p>Using appropriate weighting helps to prefer the correct node, however, when enough Dask workers are requested, a number of those workers may end up on the <code>lgworker</code> nodes. The fix to this would be to update the <code>kube-scheduler</code> to consider 100% of nodes when considering scheduling. By default the <code>kube-scheduler</code> will consider <a href="https://v1-21.docs.kubernetes.io/docs/concepts/scheduling-eviction/scheduler-perf-tuning/#default-threshold" rel="nofollow noreferrer">N% (dynamically determined)</a> of nodes at a time to evaluate by filtering and scoring <a href="https://v1-21.docs.kubernetes.io/docs/concepts/scheduling-eviction/kube-scheduler/" rel="nofollow noreferrer">v1.21.Kube-Scheduler</a>.</p>
<p>NodeAffinity will only go so far and due to it's non-guarantee of enforcing affinities, this could cause pods to be scheduled on unpreferred nodes.</p>
<p><a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.21/#nodeaffinity-v1-core" rel="nofollow noreferrer">Node Affinity v1</a> :</p>
<blockquote>
<p>The scheduler will prefer to schedule pods to nodes that satisfy the affinity expressions specified by this field, but it may choose a node that violates one or more of the expressions. The node that is most preferred is the one with the greatest sum of weights, i.e. for each node that meets all of the scheduling requirements (resource request, requiredDuringScheduling affinity expressions, etc.), compute a sum by iterating through the elements of this field and adding &quot;weight&quot; to the sum if the node matches the corresponding matchExpressions; the node(s) with the highest sum are the most preferred.</p>
</blockquote>
<pre class="lang-yaml prettyprint-override"><code>        extraPodConfig:
          tolerations:
            - key: node_toleration
              value: worker
              operator: Equal
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                - matchExpressions:
                  - key: node_label
                    operator: In
                    values:
                      - worker
                      - lgworker
              preferredDuringSchedulingIgnoredDuringExecution:
                - weight: 100
                  preference:
                    matchExpressions:
                    - key: node_label
                      operator: In
                      values: 
                      - worker
                - weight: 1
                  preference:
                    matchExpressions:
                    - key: node_label
                      operator: In
                      values: 
                      - lgworker
</code></pre>
<p>So, affecting the <code>kube-scheduler</code> would involve updating its configuration:
<a href="https://v1-21.docs.kubernetes.io/docs/concepts/scheduling-eviction/scheduler-perf-tuning/#example" rel="nofollow noreferrer">Example</a></p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: kubescheduler.config.k8s.io/v1alpha1
kind: KubeSchedulerConfiguration
algorithmSource:
  provider: DefaultProvider

...

percentageOfNodesToScore: 100
</code></pre>
