<p>I'v setup Kubernetes Horizontal Pod Autoscaler with custom metrics using the prometheus adapter <a href="https://github.com/DirectXMan12/k8s-prometheus-adapter" rel="nofollow noreferrer">https://github.com/DirectXMan12/k8s-prometheus-adapter</a>. Prometheus is monitoring rabbitmq, and Im watching the <strong>rabbitmq_queue_messages</strong> metric. The messages from the queue are picked up by the pods, that then do some processing, which can last for several hours.</p>

<p>The scale-up and scale-down is working based on the number of messages in the queue.</p>

<p><strong>The problem:</strong>
When a pod finishes the processing and acks the message, that will lower the num. of messages in the queue, and that would trigger the Autoscaler  terminate a pod. If I have multipe pods doing the processing and one of them finishes, if Im not mistaking, Kubernetes could terminate a pod that is still doing the processing of its own message. This wouldnt be desirable as all the processing that the pod is doing would be lost.</p>

<p>Is there a way to overcome this, or another way how this could be acheveed?</p>

<p>here is the Autoscaler configuration:</p>

<pre><code>kind: HorizontalPodAutoscaler
apiVersion: autoscaling/v2beta1
metadata:
  name: sample-app-rabbitmq
  namespace: monitoring
spec:
  scaleTargetRef:
    # you created above
    apiVersion: apps/v1
    kind: Deployment
    name: sample-app
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Object
    object:
      target:
        kind: Service
        name: rabbitmq-cluster
      metricName: rabbitmq_queue_messages_ready
      targetValue: 5
</code></pre>

Zarko
<p>You could consider approach using <strong>preStop hook</strong>.</p>

<p>As per documentation <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-states" rel="nofollow noreferrer">Container States</a>, <a href="https://kubernetes.io/docs/tasks/configure-pod-container/attach-handler-lifecycle-event/#define-poststart-and-prestop-handlers" rel="nofollow noreferrer">Define postStart and preStop handlers</a>:</p>

<blockquote>
  <p>Before a container enters into Terminated, preStop hook (if any) is executed.</p>
</blockquote>

<p>So you can use in your deployment:</p>

<pre><code>lifecycle:
      preStop:
        exec:
          command: ["your script"]
</code></pre>

<p><code>###</code> <strong>update</strong>:</p>

<ol>
<li><p>I would like to provide more information due to some research:
There is an interesting <a href="https://github.com/kedacore/keda" rel="nofollow noreferrer">project</a>: </p>

<blockquote>
  <p>KEDA allows for fine grained autoscaling (including to/from zero) for event driven Kubernetes workloads. KEDA serves as a Kubernetes Metrics Server and allows users to define autoscaling rules using a dedicated Kubernetes custom resource definition.
  KEDA can run on both the cloud and the edge, integrates natively with Kubernetes components such as the Horizontal Pod Autoscaler, and has no external dependencies.</p>
</blockquote></li>
<li><p>For the main question "Kubernetes could terminate a pod that is still doing the processing of its own message". </p>

<p>As per documentation:</p>

<blockquote>
  <p>"Deployment is a higher-level concept that manages ReplicaSets and provides declarative updates to Pods along with a lot of other useful features"</p>
</blockquote></li>
</ol>

<p>Deployment is backed by Replicaset. As per this controller code there exist function  "<a href="https://github.com/kubernetes/kubernetes/blob/f794c824b1e6e68b302d94f42f60af4759e18c6d/pkg/controller/replicaset/replica_set.go#L684:6" rel="nofollow noreferrer">getPodsToDelete</a>". In combination with "<em>filteredPods</em>" it gives the result: "<strong>This ensures that we delete pods in the earlier stages whenever possible.</strong>"</p>

<p>So as proof of concept:</p>

<p>You can create deployment with <strong>init container</strong>. Init container should check if there is a message in the queue and exit when at least one message appears. This will allow main container to start, take and process that message. In this case we will have <strong>two kinds of pods</strong> - those which <strong>process the message</strong> and consume CPU and those who are in the <strong>starting state</strong>, idle and waiting for the next message. In this case <strong>starting containers will be deleted at the first place</strong> when HPA decide to decrease number of replicas in the deployment. </p>

<pre><code>apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: complete
  name: complete
spec:
  replicas: 5
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: complete
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: complete
    spec:
      hostname: c1
      containers:
      - name: complete
        command: 
        - "bash"
        args:
        - "-c"
        - "wa=$(shuf -i 15-30 -n 1)&amp;&amp; echo $wa &amp;&amp; sleep $wa"
        image: ubuntu
        imagePullPolicy: IfNotPresent
        resources: {}
      initContainers:
      - name: wait-for
        image: ubuntu
        command: ['bash', '-c', 'sleep 30']


  dnsPolicy: ClusterFirst
  restartPolicy: Always
  terminationGracePeriodSeconds: 30
</code></pre>

<p>Hope this help.</p>
