<p>I'v setup Kubernetes Horizontal Pod Autoscaler with custom metrics using the prometheus adapter <a href="https://github.com/DirectXMan12/k8s-prometheus-adapter" rel="nofollow noreferrer">https://github.com/DirectXMan12/k8s-prometheus-adapter</a>. Prometheus is monitoring rabbitmq, and Im watching the <strong>rabbitmq_queue_messages</strong> metric. The messages from the queue are picked up by the pods, that then do some processing, which can last for several hours.</p>

<p>The scale-up and scale-down is working based on the number of messages in the queue.</p>

<p><strong>The problem:</strong>
When a pod finishes the processing and acks the message, that will lower the num. of messages in the queue, and that would trigger the Autoscaler  terminate a pod. If I have multipe pods doing the processing and one of them finishes, if Im not mistaking, Kubernetes could terminate a pod that is still doing the processing of its own message. This wouldnt be desirable as all the processing that the pod is doing would be lost.</p>

<p>Is there a way to overcome this, or another way how this could be acheveed?</p>

<p>here is the Autoscaler configuration:</p>

<pre><code>kind: HorizontalPodAutoscaler
apiVersion: autoscaling/v2beta1
metadata:
  name: sample-app-rabbitmq
  namespace: monitoring
spec:
  scaleTargetRef:
    # you created above
    apiVersion: apps/v1
    kind: Deployment
    name: sample-app
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Object
    object:
      target:
        kind: Service
        name: rabbitmq-cluster
      metricName: rabbitmq_queue_messages_ready
      targetValue: 5
</code></pre>

Zarko
<p>Horizontal Pod Autoscaler is not designed for long-running tasks, and will not be a good fit. If you need to spawn one long-running processing tasks per message, I'd take one of these two approaches:</p>

<ul>
<li>Use a task queue such as <a href="http://www.celeryproject.org/" rel="nofollow noreferrer">Celery</a>. It is designed to solve your exact problem: have a queue of tasks that needs to be distributed to workers, and ensure that the tasks run to completion. Kubernetes even provides an <a href="https://kubernetes.io/docs/tasks/job/coarse-parallel-processing-work-queue/" rel="nofollow noreferrer">official example</a> of this setup.</li>
<li>If you don't want to introduce another component such as Celery, you can spawn a Kubernetes <a href="https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/" rel="nofollow noreferrer">job</a> for every incoming message by yourself. Kubernetes will make sure that the job runs to completion at least once - reschedule the pod if it dies, etc. In this case you will need to write a script that reads RabbitMQ messages and creates jobs for them by yourself.</li>
</ul>

<p>In both cases, make sure you also have <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-autoscaler" rel="nofollow noreferrer">Cluster Autoscaler</a> enabled so that new nodes get automatically provisioned if your current nodes are not sufficient to handle the load.</p>
