<p>In one of my project, we have 9 PODs for a single microservice, and during the load test run that is noticing the load distribution across the 9 PODs are not even. Also, on the PODs (which have received low traffic compared to other PODs) there is a gap between the traffics. Has anyone faced this issue and advise the areas / spaces that could cause this</p>
<p>All 9 PODs are hosted on a different node under the same cluster and we have 3 zones</p>
<p>The load balancer algorithm used is round-robin.</p>
<p>Sample flow: microservices 1 (is running in 3 PODs, which uses Nginx but not as a load balancer) -&gt; microservices 2 (is running 9 PODs, which uses node js)</p>
<p>Another flow: microservices 1 (is running in 6 PODs) -&gt; microservices 2 (running in 9 PODs)</p>
<p>Refer to the below screenshots,</p>
<p><a href="https://i.stack.imgur.com/8xuNb.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/8xuNb.png" alt="Load distribution numbers across 9 PODs" /></a></p>
<p><a href="https://i.stack.imgur.com/112zJ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/112zJ.png" alt="Traffic at POD 6" /></a>[<img src="https://i.stack.imgur.com/TQ0ah.png" alt="Traffic at POD 4Traffic at POD 1/2" /></p>
<p><a href="https://i.stack.imgur.com/gkaAf.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/gkaAf.png" alt="Traffic at POD 4" /></a></p>
<p><a href="https://i.stack.imgur.com/TQ0ah.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/TQ0ah.png" alt="Traffic at POD 1/2" /></a></p>

ArunPrasath
<p>As far as Kubernetes is concerned, the LB distributes requests at the node level and not at the pod and it will completely disregard the number of pods on each node. Unfortunately, this is a limitation on Kubernetes. You may also have a look at the last paragraph of this documentation about traffic not equally load balanced across pods. [1]</p>
<p>Defining resources for containers [2] is important as it allows the scheduler to make better decisions when it comes time to place pods in nodes. May I recommend to have a look at the following documentation [3] on how pods with resource limits are set. It is mentioned that a pod will not be allowed to exceed its CPU limit for an extended period of time and it will not be killed, eventually leading to a decreased performance.</p>
<p>[1] <a href="https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer" rel="nofollow noreferrer">https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer</a></p>
<p>[2] <a href="https://kubernetes.io/docs/concepts/configuration/manage-resources-containers" rel="nofollow noreferrer">https://kubernetes.io/docs/concepts/configuration/manage-resources-containers</a></p>
<p>[3]<a href="https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#how-pods-with-resource-limits-are-run" rel="nofollow noreferrer">https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#how-pods-with-resource-limits-are-run</a></p>
<p>Regards,
Anbu.</p>
