<p>Still somewhat of a K8s noob, hoping someone can provide some guidance here.</p>
<h2>Setup:</h2>
<ul>
<li>a k8s cluster that is responsible for running CI jobs. These jobs can run for up to 20min.</li>
<li>There is an agent pod running in the cluster as well that is responsible for claiming jobs and spinning up pods in which to run them (1 pod per job instance)</li>
<li>There is a cluster-autoscaler pod running in the cluster. Due to the cyclical/variable nature of CI, we need to be able to dynamically scale the number of available nodes to meet demand.</li>
</ul>
<h2>Problem #1:</h2>
<ul>
<li>During scale-down, we are seeing job pods getting killed mid-job. This is obviously not desirable.</li>
</ul>
<h2>Attempted Solution:</h2>
<ul>
<li>I've configured the CA's <code>scale-down-utilization-threshold</code> to <code>0</code>, per the <a href="https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#how-can-i-prevent-cluster-autoscaler-from-scaling-down-non-empty-nodes" rel="nofollow noreferrer">CA FAQ</a></li>
<li>I've also shortened the <code>scale-down-unneeded-time</code> to from <code>10m</code> to <code>1m</code> in hopes of increasing the probability of an empty node staying empty long enough to trigger a scale-down.</li>
</ul>
<h2>Problem #2:</h2>
<ul>
<li>While the above does seem to solve problem #1, I now have a new issue that cluster resource utilization is much lower.</li>
</ul>
<h2>Why is this happening?</h2>
<p>I believe that by default, k8s workload placement is something like round robin between all eligible nodes (correct me if I'm wrong). This means that once the cluster scales up, there is a very high probability that a  node will be running at least 1 job. This will prevent CA from scaling down nodes, even if utilization is low and the job(s) could easily be packed into a smaller number of nodes.</p>
<h2>Desired Behavior</h2>
<p>I'm looking for a way to configure k8s to place pods in such a way that will be more conducive to higher resource utilization, given the other constraints of my cluster's configuration.</p>
<p>The only way I've been able to come up with so far would involve multiple tiers of nodes with configured labels + node affinity s.t tier 1 would fill up before tier 2 starts receiving pods, etc. While I think this would work, it seems messy and like there should be a better way.</p>

Ian Hansborough
<p>You need to take a look at <a href="https://karpenter.sh" rel="nofollow noreferrer">Karpenter</a>. Compared to the cluster-autoscaler, it operates with EC2 instances directly, not via ASG.</p>
<p>In your case, when you have a new CI task, it will create an instance that fits only required PODs, giving you more chances that no other PODs will be run on it. As a result, when work is done, the complete node will be terminated. We have used it on all our projects for over a year, and it works great.</p>
