<p>I'm trying to spread my <code>ingress-nginx-controller</code> pods such that:</p>
<ul>
<li>Each availability zone has the same # of pods (+- 1).</li>
<li>Pods prefer Nodes that currently run the least pods.</li>
</ul>
<p>Following other questions here, I have set up Pod Topology Spread Constraints in my pod deployment:</p>
<pre><code>      replicas: 4
      topologySpreadConstraints:
      - labelSelector:
          matchLabels:
            app.kubernetes.io/name: ingress-nginx
        maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
      - labelSelector:
          matchLabels:
            app.kubernetes.io/name: ingress-nginx
        maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: DoNotSchedule
</code></pre>
<p>I currently have 2 Nodes, each in a different availability zone:</p>
<pre><code>$ kubectl get nodes --label-columns=topology.kubernetes.io/zone,kubernetes.io/hostname
NAME                            STATUS   ROLES                  AGE    VERSION   ZONE         HOSTNAME
ip-{{node1}}.compute.internal   Ready    node                   136m   v1.20.2   us-west-2a   ip-{{node1}}.compute.internal
ip-{{node2}}.compute.internal   Ready    node                   20h    v1.20.2   us-west-2b   ip-{{node2}}.compute.internal

</code></pre>
<p>After running <code>kubectl rollout restart</code> for that deployment, I get 3 pods in one Node, and 1 pod in the other, which has a skew of <code>2 &gt; 1</code>:</p>
<pre><code>$ kubectl describe pod ingress-nginx-controller -n ingress-nginx | grep 'Node:'
Node:         ip-{{node1}}.compute.internal/{{node1}}
Node:         ip-{{node2}}.compute.internal/{{node2}}
Node:         ip-{{node1}}.compute.internal/{{node1}}
Node:         ip-{{node1}}.compute.internal/{{node1}}
</code></pre>
<p>Why is my constraint not respected? How can I debug the pod scheduler?</p>
<p>My kubectl version:</p>
<pre><code>$ kubectl version
Client Version: version.Info{Major:&quot;1&quot;, Minor:&quot;21+&quot;, GitVersion:&quot;v1.21.0-beta.0.607+269d62d895c297&quot;, GitCommit:&quot;269d62d895c29743931bfaaec6e8d37ced43c35f&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2021-03-05T22:28:02Z&quot;, GoVersion:&quot;go1.16&quot;, Compiler:&quot;gc&quot;, Platform:&quot;darwin/arm64&quot;}
Server Version: version.Info{Major:&quot;1&quot;, Minor:&quot;20&quot;, GitVersion:&quot;v1.20.2&quot;, GitCommit:&quot;faecb196815e248d3ecfb03c680a4507229c2a56&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2021-01-13T13:20:00Z&quot;, GoVersion:&quot;go1.15.5&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}
</code></pre>

<p>In Kubernetes 1.25, you can now use the alpha feature <code>matchLabelKeys</code> to resolve this issue. Because there is an automatically generated <code>pod-template-hash</code> for each version of a deployment added to each pod, that and your app specific label can provide the hashed value to prevent mis-scheduling.</p>
<pre><code>topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: kubernetes.io/hostname
      whenUnsatisfiable: DoNotSchedule
      matchLabelKeys:
        - app
        - pod-template-hash
</code></pre>
<blockquote>
<p>matchLabelKeys is a list of pod label keys to select the pods over
which spreading will be calculated. The keys are used to lookup values
from the pod labels, those key-value labels are ANDed with
labelSelector to select the group of existing pods over which
spreading will be calculated for the incoming pod. Keys that don't
exist in the pod labels will be ignored. A null or empty list means
only match against the labelSelector.</p>
<p>With matchLabelKeys, users don't need to update the pod.spec between
different revisions. The controller/operator just needs to set
different values to the same label key for different revisions. The
scheduler will assume the values automatically based on
matchLabelKeys. For example, if users use Deployment, they can use the
label keyed with pod-template-hash, which is added automatically by
the Deployment controller, to distinguish between different revisions
in a single Deployment.</p>
<p>Note: The matchLabelKeys field is an alpha field added in 1.25. You
have to enable the MatchLabelKeysInPodTopologySpread feature gate in
order to use it.</p>
</blockquote>
<p>From the <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/" rel="nofollow noreferrer">docs</a></p>
