<p>I have configured my ingress controller with nginx-ingress <a href="https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#custom-nginx-upstream-hashing" rel="nofollow noreferrer">hashing</a> and I define HPA for my deployments. When we do load testing we hit a problem on the newly created pods that aren't warmed up enough and while the load balancing shifts immediately target portion of the traffic the latency spikes and service is choking. Is there a way to define some smooth load rebalancing that would rather move the traffic gradually and thus warm up the service in more natural way ?</p>

<p>Here is an example effect we see now:
<a href="https://i.stack.imgur.com/5MVT6.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/5MVT6.png" alt="enter image description here"></a></p>

<p>At glance I see 2 possible reasons for that behaviour:  </p>

<ol>
<li><p>I think there is a chance that you are facing the same problem as encountered in this question: <a href="https://stackoverflow.com/q/56899429/8971507">Some requests fails during autoscaling in kubernetes</a>. In that case, Nginx was sending requests to Pods that were not completely ready. To solve this you can configure a <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/" rel="nofollow noreferrer">Readiness Probe</a>. Personally, I configure my Readiness Probes to send a http request to a /health endpoint of my services.  </p></li>
<li><p>There is a chance however that your application naturally performs slowly during the first requests, usually because of caching or some other operation that needs to be done at the beginning of its life. I encountered this problem in a Django+Gunicorn app where the Gunicorn only started my app after the first request. To solve this I used a <code>PostStart</code> <a href="https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks" rel="nofollow noreferrer">Container Hook</a> which sends a request to my app right after the container is created. <a href="https://kubernetes.io/docs/tasks/configure-pod-container/attach-handler-lifecycle-event/" rel="nofollow noreferrer">Here</a> is an example of its use. You may also have a look at this question: <a href="https://stackoverflow.com/q/57837018/8971507">Kubernetes Pod warm-up for load balancing</a>.</p></li>
</ol>
