<p>We plan to use AWS EKS to run a stateless application.</p>
<p>There is a goal to achieve optimal budget by using spot instances and prefer them to on-demand ones.</p>
<p>Per <a href="https://aws.github.io/aws-eks-best-practices/cluster-autoscaling/cluster-autoscaling/#spot-instances" rel="nofollow noreferrer">AWS recommendations</a>, we plan to have two Managed Node Groups: one with on-demand instances, and one with spot instances, plus Cluster Autoscaler to adjust groups size.</p>
<p>Now, the problem to solve is achieving two somewhat conflicting requirements:</p>
<ul>
<li>Prefer spot nodes to on-demand, e.g. run 90% of pods on spot instances and 10% on on-demand ones</li>
<li>But still, ensure that some pods always do run within on-demand group, so even in case of massive spot instance drain, there still will be some pods that can process requests</li>
</ul>
<p>After some research I found following possible approaches to solving it:</p>
<p>Approach A: Using <code>preferredDuringSchedulingIgnoredDuringExecution</code> with weights based on Node Group capacity type label. E.g. one <code>preferredDuringSchedulingIgnoredDuringExecution</code> rule with weight 90 would prefer nodes with capacity type <code>spot</code>, and other rule with weight 1 would prefer on-demand ones, e.g.:</p>
<pre><code>preferredDuringSchedulingIgnoredDuringExecution:
  - weight: 90
    preference:
      matchExpressions:
        - key: eks.amazonaws.com/capacityType
          operator: In
          values:
            - spot
  - weight: 1
    preference:
      matchExpressions:
        - key: eks.amazonaws.com/capacityType
          operator: NotIn
          values:
            - spot
</code></pre>
<p>The downside is that — as I understand — you are not guaranteed to have pods running on least preferred group, as those are just (added) weights, not some sort of exact distribution.</p>
<p>Other approach, which in theory could be combined with one above (?) is also using <code>topologySpreadConstraints</code>, e.g.:</p>
<pre><code>spec:
  topologySpreadConstraints:
  - maxSkew: 20
    topologyKey: eks.amazonaws.com/capacityType
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        foo: bar
</code></pre>
<p>Which would distribute pods across nodes with different capacity types, while allowing a skew of, say, 20 pods between them, and probably should (?) be combined with <code>preferredDuringSchedulingIgnoredDuringExecution</code> to achieve the desired effect.</p>
<p>How feasible is the approach above? Are those the right tools to achieve the goals? I would very much appreciate any advice on the case!</p>

dusty
<p>This is not something the Kubernetes scheduler supports. Weights in affinities are more like score multiplies, and maxSkew is a very general cap on how out of balance things can get, but not the direction of that imbalance.</p>
<p>You would have to write something custom AFAIK, or at least I've not seen anything for this when I went looking last. Check out the scheduler extender webhook system for a somewhat easy way to implement it.</p>
