<p>I've got a batch job to run: process a large number of media files. I have a Kubernetes cluster to run it on, but I don't want to change the size of the cluster. I want to run the processing as a low-priority job. Any time there are spare compute resources, they should work on media-processing. Any time there are other jobs that need resources, the media process should be suspended.</p>
<p>Currently, I'm running a <a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#deployment-v1-apps" rel="nofollow noreferrer">Deployment</a> with one replica for each node in my cluster. I defined a <a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#priorityclass-v1-scheduling-k8s-io" rel="nofollow noreferrer">PriorityClass</a> for the batch-job and a different PriorityClass (with higher priority) for everything else. That seems to be working to evict running batch-jobs when something else needs the resources.</p>
<p>I define a <a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#affinity-v1-core" rel="nofollow noreferrer">Affinity</a>, specifically a <a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#weightedpodaffinityterm-v1-core" rel="nofollow noreferrer">WeightedPod(Anti)Affinity</a> to discourage the batch-job from scheduling on the same machine.</p>
<p>The code itself is a queue-worker: it pulls one work-item off a shared queue and processes it and then goes back for the next. If it gets interrupted (because it's being evicted) the partial work is lost (which is fine).</p>
<p>This is working OK, but I'm leaving a lot of resources on the table, still. Is there some way to define my replica-count as &quot;as many as you can schedule&quot;? I could ask for far more replicas than the cluster can handle; would that be a good solution? Or are there problems with Kubernetes having 10 pods stuck &quot;pending&quot; for months at a time?</p>

<p>I think there's no harm in asking for more pods than the cluster can handle and keeping them pending forever. My only concern is whether the scheduler will be able to discern normal priority pending pods over low priority pending pods, and be able to give precedence to the more urgent ones.</p>
<p>The pro way to go about this issue, IMHO, is to leverage <a href="https://github.com/DirectXMan12/k8s-prometheus-adapter" rel="nofollow noreferrer">prometheus adapter</a> and use an HPA to target the current capacity of your cluster using a prometheus query. This can give you continuous of the cluster capacity and the ability to autoscale accordingly. <a href="https://medium.com/uptime-99/kubernetes-hpa-autoscaling-with-custom-and-external-metrics-da7f41ff7846" rel="nofollow noreferrer">This medium article</a> has a pretty good introduction to the concept.</p>
