<p>Here is what I am working with. 
I have 3 nodepools on GKE </p>

<ol>
<li>n1s1 (3.75GB)</li>
<li>n1s2 (7.5GB) </li>
<li>n1s4 (15GB)</li>
</ol>

<p>I have pods that will require any of the following memory requests. Assume limits are very close to requests. </p>

<pre><code>1GB, 2GB, 4GB, 6GB, 8GB, 10GB, 12GB, 14GB
</code></pre>

<p>How best can I associate a pod to a nodepool for max efficiency?</p>

<p>So far I have 3 strategies.</p>

<p>For each pod config, determine the <strong><em>“rightful nodepool”</em></strong>. This is the smallest nodepool that can accommodate the pod config in an ideal world.
So for 2GB pod it's n1s1 but for 4GB pod it'd be n1s2.</p>

<ol>
<li>Schedule a pod only on its <em>rightful nodepool.</em> </li>
<li>Schedule a pod only on its <em>rightful nodepool</em> or one nodepool higher than that.</li>
<li>Schedule a pod only on any nodepool where it can currently go. </li>
</ol>

<p>Which of these or any other strategies will minimize wasting resources?</p>

<p>=======</p>

<p>I would use a mix of <a href="https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/" rel="nofollow noreferrer">Taints and Tolerations</a> and <a href="https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#node-affinity" rel="nofollow noreferrer">Node affinity</a>.</p>

<blockquote>
  <p>Taints and tolerations work together to ensure that pods are not scheduled onto inappropriate nodes. One or more taints are applied to a node; this marks that the node should not accept any pods that do not tolerate the taints. Tolerations are applied to pods, and allow (but do not require) the pods to schedule onto nodes with matching taints.</p>
</blockquote>

<p>You can set a taint on a node <code>kubectl taint nodes node1 key=value:NoSchedule</code></p>

<blockquote>
  <p>The taint has key <code>key</code>, value <code>value</code>, and taint effect <code>NoSchedule</code>. This means that no pod will be able to schedule onto <code>node1</code> unless it has a matching toleration.</p>
</blockquote>

<p>While you are writing a <code>pod</code> yaml you can specify <code>PodSpec</code> and add toleration which will match the taint created on <code>node1</code> which will allow <code>pod</code> with either toleration to be scheduled onto <code>node1</code></p>

<pre class="lang-yaml prettyprint-override"><code>tolerations:
- key: "key"
  operator: "Equal"
  value: "value"
  effect: "NoSchedule"
</code></pre>

<p>or</p>

<pre class="lang-yaml prettyprint-override"><code>tolerations:
- key: "key"
  operator: "Exists"
  effect: "NoSchedule"
</code></pre>

<blockquote>
  <p>Taints and tolerations are a flexible way to steer pods  <em>away</em>  from nodes or evict pods that shouldn’t be running. A few of the use cases are</p>
  
  <ul>
  <li><p><strong>Dedicated Nodes</strong>: If you want to dedicate a set of nodes for exclusive use by a particular set of users, you can add a taint to those nodes (say,  <code>kubectl taint nodes nodename dedicated=groupName:NoSchedule</code>) and then add a corresponding toleration to their pods (this would be done most easily by writing a custom  <a href="https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/" rel="nofollow noreferrer">admission controller</a>). The pods with the tolerations will then be allowed to use the tainted (dedicated) nodes as well as any other nodes in the cluster. If you want to dedicate the nodes to them  <em>and</em>  ensure they  <em>only</em>  use the dedicated nodes, then you should additionally add a label similar to the taint to the same set of nodes (e.g.  <code>dedicated=groupName</code>), and the admission controller should additionally add a node affinity to require that the pods can only schedule onto nodes labeled with  <code>dedicated=groupName</code>.</p></li>
  <li><p><strong>Nodes with Special Hardware</strong>: In a cluster where a small subset of nodes have specialized hardware (for example GPUs), it is desirable to keep pods that don’t need the specialized hardware off of those nodes, thus leaving room for later-arriving pods that do need the specialized hardware. This can be done by tainting the nodes that have the specialized hardware (e.g.  <code>kubectl taint nodes nodename special=true:NoSchedule</code>  or  <code>kubectl taint nodes nodename special=true:PreferNoSchedule</code>) and adding a corresponding toleration to pods that use the special hardware. As in the dedicated nodes use case, it is probably easiest to apply the tolerations using a custom  <a href="https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/" rel="nofollow noreferrer">admission controller</a>. For example, it is recommended to use  <a href="https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#extended-resources" rel="nofollow noreferrer">Extended Resources</a>  to represent the special hardware, taint your special hardware nodes with the extended resource name and run the  <a href="https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#extendedresourcetoleration" rel="nofollow noreferrer">ExtendedResourceToleration</a>  admission controller. Now, because the nodes are tainted, no pods without the toleration will schedule on them. But when you submit a pod that requests the extended resource, the  <code>ExtendedResourceToleration</code>  admission controller will automatically add the correct toleration to the pod and that pod will schedule on the special hardware nodes. This will make sure that these special hardware nodes are dedicated for pods requesting such hardware and you don’t have to manually add tolerations to your pods.</p></li>
  <li><p><strong>Taint based Evictions</strong>: A per-pod-configurable eviction behavior when there are node problems, which is described in the next section.</p></li>
  </ul>
</blockquote>

<p>As for <a href="https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#node-affinity" rel="nofollow noreferrer"><code>node affinity</code></a>:</p>

<blockquote>
  <p>is conceptually similar to <code>nodeSelector</code> – it allows you to constrain which nodes your pod is eligible to be scheduled on, based on labels on the node.</p>
  
  <p>There are currently two types of node affinity, called  <code>requiredDuringSchedulingIgnoredDuringExecution</code>  and  <code>preferredDuringSchedulingIgnoredDuringExecution</code>. You can think of them as “hard” and “soft” respectively, in the sense that the former specifies rules that  <em>must</em>  be met for a pod to be scheduled onto a node (just like  <code>nodeSelector</code>  but using a more expressive syntax), while the latter specifies  <em>preferences</em>  that the scheduler will try to enforce but will not guarantee. The “IgnoredDuringExecution” part of the names means that, similar to how  <code>nodeSelector</code>  works, if labels on a node change at runtime such that the affinity rules on a pod are no longer met, the pod will still continue to run on the node. In the future we plan to offer  <code>requiredDuringSchedulingRequiredDuringExecution</code>  which will be just like  <code>requiredDuringSchedulingIgnoredDuringExecution</code>  except that it will evict pods from nodes that cease to satisfy the pods’ node affinity requirements.</p>
  
  <p>Thus an example of  <code>requiredDuringSchedulingIgnoredDuringExecution</code>  would be “only run the pod on nodes with Intel CPUs” and an example  <code>preferredDuringSchedulingIgnoredDuringExecution</code>  would be “try to run this set of pods in failure zone XYZ, but if it’s not possible, then allow some to run elsewhere”.</p>
  
  <p>Node affinity is specified as field  <code>nodeAffinity</code>  of field  <code>affinity</code>  in the PodSpec.</p>
  
  <p>...</p>
  
  <p>The new node affinity syntax supports the following operators:  <code>In</code>,  <code>NotIn</code>,  <code>Exists</code>,  <code>DoesNotExist</code>,  <code>Gt</code>,  <code>Lt</code>. You can use  <code>NotIn</code>  and  <code>DoesNotExist</code>  to achieve node anti-affinity behavior, or use  <a href="https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/" rel="nofollow noreferrer">node taints</a>  to repel pods from specific nodes.</p>
  
  <p>If you specify both  <code>nodeSelector</code>  and  <code>nodeAffinity</code>,  <em>both</em>  must be satisfied for the pod to be scheduled onto a candidate node.</p>
  
  <p>If you specify multiple  <code>nodeSelectorTerms</code>  associated with  <code>nodeAffinity</code>  types, then the pod can be scheduled onto a node  <strong>only if all</strong>  <code>nodeSelectorTerms</code>  can be satisfied.</p>
  
  <p>If you specify multiple  <code>matchExpressions</code>  associated with  <code>nodeSelectorTerms</code>, then the pod can be scheduled onto a node  <strong>if one of</strong>  the  <code>matchExpressions</code>  is satisfied.</p>
</blockquote>
