<h2>Overview</h2>
<p>Kubernetes scheduling errs on the side of 'not shuffling things around once scheduled and happy' which can lead to quite the level of imbalance in terms of CPU, Memory, and container count distribution. It can also mean that <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/#known-limitations" rel="nofollow noreferrer">sometimes Affinity and Topology rules may not be enforced</a> / as the state of affair changes:</p>
<h6>With regards to topology spread constraints introduced in v1.19 (stable)</h6>
<blockquote>
<p>There's no guarantee that the constraints remain satisfied when Pods are removed. For example, scaling down a Deployment may result in imbalanced Pods distribution.</p>
</blockquote>
<h2>Context</h2>
<p>We are currently making use of <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/" rel="nofollow noreferrer">pod topology spread contraints</a>, and they are pretty superb, aside from the fact that they only seem to handle skew during scheduling, and not execution (unlike the ability to differentiate with <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/#concepts" rel="nofollow noreferrer">Taints and Tolerations</a>).</p>
<p>For features such as <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinity" rel="nofollow noreferrer">Node affinity</a>, we're currently waiting on the ability to add <code>RequiredDuringExecution</code> requirements as opposed to <code>ScheduledDuringExecution</code> requirements</p>
<h2>Question</h2>
<p><em><strong>My question is</strong></em>, is there a native way to make Kubernetes re-evaluate and attempt to enforce topology spread skew when a new fault domain (topology) is added, without writing my own scheduler?</p>
<p>Or do I need to wait for Kubernetes to advance a few more releases? ;-) (I'm hoping someone may have a smart answer with regards to combining affinity / topology constraints)</p>

<p>After more research I'm fairly certain that using an outside tool like <a href="https://github.com/kubernetes-sigs/descheduler" rel="nofollow noreferrer">Descheduler</a> is the best way currently.</p>
<p>There doesn't seem to be a combination of Taints, Affinity rules, or Topology constraints that can work together to achieve the re-evaluation of topology rules during execution.</p>
<p>Descheduler allows you to kill of certain workloads based on user requirements, and let the default <code>kube-scheduler</code> reschedule killed pods. It can be installed easily with manifests or Helm and ran on a schedule. It can even be triggered manually when the topology changes, which is what I think we will implement to suit our needs.</p>
<p>This will be the best means of achieving our goal while waiting for <code>RequiredDuringExecution</code> rules to mature across all feature offerings.</p>
<p>Given our topology rules mark each node as a topological zone, using a <a href="https://github.com/kubernetes-sigs/descheduler/blob/master/examples/low-node-utilization.yml" rel="nofollow noreferrer">Low Node Utilization strategy</a> to spread workloads across new hosts as they appear will be what we go with.</p>
<pre><code>apiVersion: &quot;descheduler/v1alpha1&quot;
kind: &quot;DeschedulerPolicy&quot;
strategies:
  &quot;LowNodeUtilization&quot;:
    enabled: true
    params:
      nodeResourceUtilizationThresholds:
        thresholds:
          &quot;memory&quot;: 20
        targetThresholds:
          &quot;memory&quot;: 70
</code></pre>
