<p>I'm currently using AWS EBS but switching over to Kubernetes for managing our celery worker pool. According to the Celery documentation when you want to stop a celery worker there are <a href="https://docs.celeryproject.org/en/stable/userguide/workers.html#stopping-the-worker" rel="nofollow noreferrer">two types of signals that can be sent to the worker</a>. One being <strong>TERM</strong> and another being <strong>KILL</strong>. When <strong>TERM</strong> is used it waits for the worker to finish what it is currently executing before stopping. When <strong>KILL</strong> is used to stops it immediately and this <strong>can cause the loss of tasks</strong>. My question is with Kubernetes autoscaling when scaling down the worker pool, how do I ensure TERM is sent to the workers rather than KILL? I was experiencing this scale down problem with AWS EBS also where Celery flower was showing the loss of tasks when scaling down.</p>

jjdoherty
<p>The chain of events here is:</p>
<ol>
<li>The HorizontalPodAutoscaler decreases the <code>replicas:</code> on the Deployment it controls.</li>
<li>The Deployment decreases the <code>replicas:</code> on its corresponding ReplicaSet.</li>
<li>The ReplicaSet deletes Pods as required.</li>
<li>Kubernetes sends each container SIGTERM and waits for it to shut down.</li>
<li>If the container hasn't already exited, it sends each container SIGKILL.</li>
</ol>
<p>The shutdown sequence in particular is described in <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination" rel="nofollow noreferrer">Termination of Pods</a>.  Your process will get SIGTERM, and then SIGKILL 30 seconds later if it hasn't already exited.  Nothing is special about the HPA deleting a pod as opposed to any other path.</p>
