<blockquote>
<p><strong>TL;DR: NodeSelector ignores nodes from another NodePool. How to distribute pods across more NodePools using a label nodeSelector or other technique?</strong></p>
</blockquote>
<p>I have two nodePools like this:</p>
<pre class="lang-golang prettyprint-override"><code>...
# Spot node pool
resource &quot;azurerm_kubernetes_cluster_node_pool&quot; &quot;aks_staging_np_compute_spot&quot; {
  name                  = &quot;computespot&quot;
  (...)
  vm_size               = &quot;Standard_F8s_v2&quot;
  max_count             = 2
  min_count             = 2
  (...)
  priority = &quot;Spot&quot;
  eviction_policy = &quot;Delete&quot;
  (...)
  node_labels = {
    &quot;pool_type&quot; = &quot;compute&quot;
  }

# Regular node pool
resource &quot;azurerm_kubernetes_cluster_node_pool&quot; &quot;aks_staging_np_compute_base&quot; {
  name                  = &quot;computebase&quot;
  (...)
  vm_size               = &quot;Standard_F8s_v2&quot;
  max_count             = 2
  min_count             = 2
  node_labels = {
    &quot;pool_type&quot; = &quot;compute&quot;
  }
</code></pre>
<p>Both pools are deployed in AKS and all the nodes are present in OK state. Please note two things:</p>
<ul>
<li>Both have label <code>pool_type: compute</code></li>
<li>Both have same size as <code>Standard_F8s_v2</code></li>
</ul>
<p>(There are also 20 other nodes with different labels in my cluster which are not important.)</p>
<p>Then I've got a deployment like this (omitted irrelevant lines for brevity):</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: apps/v1
kind: Deployment
metadata:
  (...)
spec:
  replicas: 4
  selector:
    matchLabels:
      app: myapp
  template:
    (...)
    spec:
      nodeSelector:
        pool_type: compute
      (...)
      containers:
        (...)
</code></pre>
<p>There is also entry in <code>tolerations</code> for accepting Azure spot instances. It apparently works.</p>
<pre class="lang-yaml prettyprint-override"><code>tolerations:
        - key: &quot;kubernetes.azure.com/scalesetpriority&quot;
          operator: &quot;Equal&quot;
          value: &quot;spot&quot;
          effect: &quot;NoSchedule&quot;
</code></pre>
<p>The problem is that <strong>the app gets deployed only on one nodepool (<code>&quot;computespot&quot;</code> in this case) and <em>never</em> touches the another (<code>computebase</code>). Even when the label and the size of the individual nodes are same.</strong></p>
<ul>
<li>2 pods are running on <code>computespot</code> nodes, one node per each.</li>
<li>The second 2 pods aren't scheduled with classic error <code>0/24 nodes are available: 14 Insufficient cpu, 17 Insufficient memory, 4 node(s) didn't match node selector.</code> <em>That's a absolute lie</em> because I can see the <code>computebase</code> nodes just sitting there entirely empty.</li>
</ul>
<p>How this can be solved?</p>

rudolfdobias
<p>Found a solution using pod affinity.</p>
<pre class="lang-yaml prettyprint-override"><code>spec:
      # This didn't work:
      #
      # nodeSelector:
      #   pool_type: compute 
      # 
      # But this does:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                - key: pool_type
                  operator: In
                  values:
                  - compute 
</code></pre>
<p>I don't know the reason because we're still dealing with one single label. If someone knows, please share.</p>
