<p>I have a kubernetes HPA set up in my cluster, and it works as expected scaling up and down instances of pods as the cpu/memory increases and decreases.</p>

<p>The only thing is that my pods handle web requests, so it occasionally scales down a pod that's in the process of handling a web request. The web server never gets a response back from the pod that was scaled down and thus the caller of the web api gets an error back.</p>

<p>This all makes sense theoretically. My question is does anyone know of a best practice way to handle this? Is there some way I can wait until all requests are processed before scaling down? Or some other way to ensure that requests complete before HPA scales down the pod?</p>

<p>I can think of a few solutions, none of which I like:</p>

<ol>
<li>Add retry mechanism to the caller and just leave the cluster as is.</li>
<li>Don't use HPA for web request pods (seems like it defeats the purpose).</li>
<li>Try to create some sort of custom metric and see if I can get that metric into Kubernetes (e.x <a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#support-for-custom-metrics" rel="nofollow noreferrer">https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#support-for-custom-metrics</a>) </li>
</ol>

<p>Any suggestions would be appreciated. Thanks in advance!</p>

<h1>Graceful shutdown of pods</h1>
<p>You must design your apps to support <em>graceful shutdown</em>. First your pod will receive a <code>SIGTERM</code> signal and after 30 seconds (can be configured) your pod will receive a <code>SIGKILL</code> signal and be removed. See <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination" rel="nofollow noreferrer">Termination of pods</a></p>
<p><strong>SIGTERM</strong>: When your app receives termination signal, your pod will not receive <strong>new requests</strong> but you should try to fulfill responses of already received requests.</p>
<h2>Design for idempotency</h2>
<p>Your apps should also be designed for <strong>idempotency</strong> so you can safely <strong>retry</strong> failed requests.</p>
