<p>Suppose I have the following nodes with labels <code>env=staging</code> and <code>env=production</code></p>
<pre><code>server0201     Ready    worker   79d   v1.18.2   10.2.2.22     &lt;none&gt;        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://19.3.5     production
server0202     Ready    worker   79d   v1.18.2   10.2.2.23     &lt;none&gt;        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://19.3.5     production
server0203     Ready    worker   35d   v1.18.3   10.2.2.30     &lt;none&gt;        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://19.3.5     staging
server0301     Ready    worker   35d   v1.18.3   10.2.3.21     &lt;none&gt;        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://19.3.5     production
server0302     Ready    worker   35d   v1.18.3   10.2.3.29     &lt;none&gt;        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://19.3.5     production
server0303     Ready    worker   35d   v1.18.0   10.2.3.30     &lt;none&gt;        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://19.3.5     staging
server0304     Ready    worker   65d   v1.18.2   10.2.6.22     &lt;none&gt;        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://19.3.5     production
</code></pre>
<p>I tried using <code>nodeSelector</code> and <code>nodeAffinity</code> but all my pods keeps residing to <code>server0203</code> and never to <code>server0303</code>, regardless how many replicas I create, when I my selector label is <code>env=staging</code>.</p>
<p>The same if I use <code>env=production</code> it will only lands server0201.</p>
<p>What should I do to make sure my pods are evenly distributed to nodes I assigned with those labels?</p>
<p>here is my deployment spec</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: helloworld
  namespace: gab
spec:
  selector:
    matchLabels:
      app: helloworld
  replicas: 2 # tells deployment to run 1 pods matching the template
  template: # create pods using pod definition in this template
    metadata:
      labels:
        app: helloworld
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: env
                operator: Equals
                values:
                - staging
      containers:
      - name: helloworld
        image: karthequian/helloworld:latest
        ports:
        - containerPort: 80
</code></pre>
<p>There are no taints in worker nodes</p>
<pre><code>kubectl get nodes -o json | jq '.items[].spec.taints'
[
  {
    &quot;effect&quot;: &quot;NoSchedule&quot;,
    &quot;key&quot;: &quot;node-role.kubernetes.io/master&quot;
  }
]
[
  {
    &quot;effect&quot;: &quot;NoSchedule&quot;,
    &quot;key&quot;: &quot;node-role.kubernetes.io/master&quot;
  }
]
[
  {
    &quot;effect&quot;: &quot;NoSchedule&quot;,
    &quot;key&quot;: &quot;node-role.kubernetes.io/master&quot;
  }
]
null
null
null
null
null
null
null
</code></pre>
<p>display of all labels</p>
<pre><code>server0201     Ready    worker   80d   v1.18.2   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,env=production,kubernetes.io/arch=amd64,kubernetes.io/hostname=eye0202,kubernetes.io/os=linux,node-role.kubernetes.io/worker=,role=worker
server0202     Ready    worker   80d   v1.18.2   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,env=production,kubernetes.io/arch=amd64,kubernetes.io/hostname=eye0203,kubernetes.io/os=linux,node-role.kubernetes.io/worker=,role=worker
server0203     Ready    worker   35d   v1.18.3   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,env=staging,kubernetes.io/arch=amd64,kubernetes.io/hostname=eye0210,kubernetes.io/os=linux,node-role.kubernetes.io/worker=,role=worker
server0301     Ready    worker   35d   v1.18.3   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,env=production,kubernetes.io/arch=amd64,kubernetes.io/hostname=eye0301,kubernetes.io/os=linux,node-role.kubernetes.io/worker=,role=worker
server0302     Ready    worker   35d   v1.18.3   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,env=production,kubernetes.io/arch=amd64,kubernetes.io/hostname=eye0309,kubernetes.io/os=linux,node-role.kubernetes.io/worker=,role=worker
server0303     Ready    worker   35d   v1.18.0   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,env=staging,kubernetes.io/arch=amd64,kubernetes.io/hostname=eye0310,kubernetes.io/os=linux,node-role.kubernetes.io/worker=,role=worker
server0304     Ready    worker   65d   v1.18.2   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,env=production,kubernetes.io/arch=amd64,kubernetes.io/hostname=eye0602,kubernetes.io/os=linux,node-role.kubernetes.io/worker=,role=worker
</code></pre>

<p>After playing for around, I realised there is no problem with <code>NodeSelector</code> or <code>PodAffinity</code>. As a matter of fact, I could even achieve what my question intends to achieve , by using <code>node-selector</code> annotations confine within my namespace.</p>
<pre><code>apiVersion: v1
kind: Namespace
metadata:
  name: gab
  annotations:
    scheduler.alpha.kubernetes.io/node-selector: env=production
spec: {}
status: {}    
</code></pre>
<p>As long as my deployment is within the namespace, the node selector works.</p>
<pre><code>kind: Deployment
metadata:
  name: helloworld
  namespace: gab
spec:
  selector:
    matchLabels:
      app: helloworld
  replicas: 10 # tells deployment to run 1 pods matching the template
  template: # create pods using pod definition in this template
    metadata:
      labels:
        app: helloworld
    spec:
      containers:
      - name: helloworld
        image: karthequian/helloworld:latest
        ports:
        - containerPort: 80
</code></pre>
<p>Now why it does work for me in the begining was that the 2nd node of my <code>staging</code> labeled nodes are slightly higher utilisation than the one I kept residing on.</p>
<pre><code>Resource           Requests     Limits
  --------           --------     ------
  cpu                3370m (14%)  8600m (35%)
  memory             5350Mi (4%)  8600Mi (6%)
  ephemeral-storage  0 (0%)       0 (0%)
</code></pre>
<p>the node I keep landing one is</p>
<pre><code>  Resource           Requests    Limits
  --------           --------    ------
  cpu                1170m (4%)  500100m (2083%)
  memory             164Mi (0%)  100Mi (0%)
  ephemeral-storage  0 (0%)      0 (0%)
</code></pre>
<p>When i test and switch to <code>production</code>, because there is more nodes, it is distributed to a few.</p>
<p>Therefore and I think, the scheduler balances pods based on <code>Server load</code> (I can be wrong) than trying to evenly distribute</p>
