<p>I set up a k8s cluster on microk8s and I ported my application to it. I also added a horizontal auto-scaler which adds pods based on the cpu load. The auto-scaler works fine and it adds pods when there is load beyond the target and when I remove the load after some time it will kill the pods.</p>

<p>The problem is I noticed at the exact same moments that the auto-scaler is creating new pods some of the requests fail:</p>

<pre><code>POST Response Code :  200
POST Response Code :  200
POST Response Code :  200
POST Response Code :  200
POST Response Code :  200
POST Response Code :  502
java.io.IOException: Server returned HTTP response code: 502 for URL: http://10.203.101.61/gateway/compile
POST Response Code :  502
java.io.IOException: Server returned HTTP response code: 502 for URL: http://10.203.101.61/gateway/compile
POST Response Code :  200
POST Response Code :  502
java.io.IOException: Server returned HTTP response code: 502 for URL: http://10.203.101.61/gateway/compile
POST Response Code :  502
java.io.IOException: Server returned HTTP response code: 502 for URL: http://10.203.101.61/gateway/compile
POST Response Code :  200
POST Response Code :  200
POST Response Code :  200
POST Response Code :  200
POST Response Code :  200
POST Response Code :  200
POST Response Code :  200
POST Response Code :  200
</code></pre>

<p>I like to know what is the reason for this and how I can fix it?</p>

<p>Update:
I think it is better I give you more information regarding my set up:</p>

<p>The traffic is coming from outside of the cluster but both the k8s node and the program that generates the requests are on one machine so there is no network problem. There is a custom nginx component which doesn't do load balancing and just act as a reverse proxy and sends the traffic to respective services.</p>

<p>I ran another test which gave me more info. I ran the same benchmarking test but this time instead of sending the requests to the reverse proxy (nginx) I used the IP address of that specific service and I had no failed request while auto-scaler did its job and launched multiple pods. I am not sure if the problem is Nginx or k8s?</p>

<p>When the new pods are spawned, Kubernetes immediately starts to redirect traffic to them.
However, usually, it takes a bit of time for the pod to boot and become operational (ready).</p>

<p>To prevent this from happening, you can define a <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#define-readiness-probes" rel="nofollow noreferrer">Readiness Probe</a> for your pods.
K8s will periodically call the pod on the readiness endpoint that you have provided, to determine if this pod is functional and ready to accept requests.
K8s won't redirect traffic to the pod until the readiness endpoints returns a successful result depending on the <a href="https://cloud.google.com/blog/products/gcp/kubernetes-best-practices-setting-up-health-checks-with-readiness-and-liveness-probes" rel="nofollow noreferrer">type of probe</a> (check "<em>Types of Probes</em>" section).</p>
