<p>I'm trying to spread my <code>ingress-nginx-controller</code> pods such that:</p>
<ul>
<li>Each availability zone has the same # of pods (+- 1).</li>
<li>Pods prefer Nodes that currently run the least pods.</li>
</ul>
<p>Following other questions here, I have set up Pod Topology Spread Constraints in my pod deployment:</p>
<pre><code>      replicas: 4
      topologySpreadConstraints:
      - labelSelector:
          matchLabels:
            app.kubernetes.io/name: ingress-nginx
        maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
      - labelSelector:
          matchLabels:
            app.kubernetes.io/name: ingress-nginx
        maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: DoNotSchedule
</code></pre>
<p>I currently have 2 Nodes, each in a different availability zone:</p>
<pre><code>$ kubectl get nodes --label-columns=topology.kubernetes.io/zone,kubernetes.io/hostname
NAME                            STATUS   ROLES                  AGE    VERSION   ZONE         HOSTNAME
ip-{{node1}}.compute.internal   Ready    node                   136m   v1.20.2   us-west-2a   ip-{{node1}}.compute.internal
ip-{{node2}}.compute.internal   Ready    node                   20h    v1.20.2   us-west-2b   ip-{{node2}}.compute.internal

</code></pre>
<p>After running <code>kubectl rollout restart</code> for that deployment, I get 3 pods in one Node, and 1 pod in the other, which has a skew of <code>2 &gt; 1</code>:</p>
<pre><code>$ kubectl describe pod ingress-nginx-controller -n ingress-nginx | grep 'Node:'
Node:         ip-{{node1}}.compute.internal/{{node1}}
Node:         ip-{{node2}}.compute.internal/{{node2}}
Node:         ip-{{node1}}.compute.internal/{{node1}}
Node:         ip-{{node1}}.compute.internal/{{node1}}
</code></pre>
<p>Why is my constraint not respected? How can I debug the pod scheduler?</p>
<p>My kubectl version:</p>
<pre><code>$ kubectl version
Client Version: version.Info{Major:&quot;1&quot;, Minor:&quot;21+&quot;, GitVersion:&quot;v1.21.0-beta.0.607+269d62d895c297&quot;, GitCommit:&quot;269d62d895c29743931bfaaec6e8d37ced43c35f&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2021-03-05T22:28:02Z&quot;, GoVersion:&quot;go1.16&quot;, Compiler:&quot;gc&quot;, Platform:&quot;darwin/arm64&quot;}
Server Version: version.Info{Major:&quot;1&quot;, Minor:&quot;20&quot;, GitVersion:&quot;v1.20.2&quot;, GitCommit:&quot;faecb196815e248d3ecfb03c680a4507229c2a56&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2021-01-13T13:20:00Z&quot;, GoVersion:&quot;go1.15.5&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}
</code></pre>

<p><code>kubectl rollout restart</code> spins up new pods and then terminate old pods <strong>after</strong> all the new pods are up and running.</p>
<p>From the pod topology spread constraints <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/#known-limitations" rel="nofollow noreferrer">known limitations</a> section, constraints don't remain satisfied when pods are removed and the recommended mitigation is now to use <a href="https://github.com/kubernetes-sigs/descheduler" rel="nofollow noreferrer">Descheduler</a> , which you already seemed to have been using from your comment.</p>
