<p>I need to scale a set of pods that run queue-based workers. Jobs for workers can run for a long time (hours) and should not get interrupted. The number of pods is based on the length of the worker queue. Scaling would be either using the horizontal autoscaler using custom metrics, or a simple controller that changes the number of replicas.</p>

<p>Problem with either solution is that, when scaling down, there is no control over which pod(s) get terminated. At any given time, most workers are likely working on short running jobs, idle, or (more rare) processing a long running job. I'd like to avoid killing the long running job workers, idle or short running job workers can be terminated without issue.</p>

<p>What would be a way to do this with low complexity? One thing I can think of is to do this based on CPU usage of the pods. Not ideal, but it could be good enough. Another method could be that workers somehow expose a priority indicating whether they are the preferred pod to be deleted. This priority could change every time a worker picks up a new job though.</p>

<p>Eventually all jobs will be short running and this problem will go away, but that is a longer term goal for now.</p>

Stragulus
<p>Since version 1.22 there is a beta feature that helps you do that. You can add the annotation <code>controller.kubernetes.io/pod-deletion-cost</code> with a value in the range <code>[-2147483647, 2147483647]</code> and this will cause pods with lower value to be killed first. Default is 0, so anything negative on one pod will cause a pod to get killed during downscaling, e.g.</p>
<p><code>kubectl annotate pods my-pod-12345678-abcde controller.kubernetes.io/pod-deletion-cost=-1000</code></p>
<p>Link to discussion about the implementation of this feature: <a href="https://github.com/kubernetes/enhancements/issues/2255" rel="nofollow noreferrer">Scale down a deployment by removing specific pods (PodDeletionCost) #2255 </a></p>
<p>Link to the documentation: <a href="https://kubernetes.io/docs/concepts/workloads/controllers/replicaset/#pod-deletion-cost" rel="nofollow noreferrer">ReplicaSet / Pod deletion cost</a></p>
