<p>I have a celery worker deployed on Kubernetes pods which executes a task (not very CPU intensive but takes some time to complete due to some HTTP calls). Is there any way to autoscale the pods in K8s based on the number of tasks in the task queue?</p>

<p>Yes, by using the Kubernetes metrics registry and <a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/" rel="noreferrer">Horizontal Pod Autoscaler</a>.</p>

<p><strong>First, you need to collect the "queue length" metric from Celery and expose it through one of the Kubernetes metric APIs. You can do this with a Prometheus-based pipeline:</strong></p>

<ol>
<li>Since Celery doesn't expose Prometheus metrics, you need to install an exporter that exposes some information about Celery (including the queue length) as Prometheus metrics. For example, <a href="https://github.com/zerok/celery-prometheus-exporter" rel="noreferrer">this exporter</a>.</li>
<li>Install <a href="https://prometheus.io/" rel="noreferrer">Prometheus</a> in your cluster and configure it to collect the metric corresponding to the task queue length from the Celery exporter.</li>
<li>Install the <a href="https://github.com/DirectXMan12/k8s-prometheus-adapter" rel="noreferrer">Prometheus Adapter</a> in your cluster and configure it to expose the "queue length" metric through the <a href="https://github.com/kubernetes/community/blob/master/contributors/design-proposals/instrumentation/custom-metrics-api.md" rel="noreferrer">Custom Metrics API</a> by pulling its value from Prometheus.</li>
</ol>

<p><strong>Now you can configure the Horizontal Pod Autoscaler to query this metric from the Custom Metrics API and autoscale your app based on it.</strong></p>

<p>For example, to scale the app between 1 and 10 replicas based on a target value for the queue length of 5:</p>

<pre><code>apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: myapp-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp
  minReplicas: 1
  maxReplicas: 10
  metrics:
    - type: Object
      object:
        metric:
          name: mycelery_queue_length
        target:
          type: value
          value: 5
        describedObject:
          apiVersion: apps/v1
          kind: Deployment
          name: mycelery
</code></pre>
