<p>I notice the cpu utilization of pods in same hpa varies from 31m to 1483m. Is this expected and normal? See below for the cpu utilization of 8 pods which are of the same hpa.</p>
<pre><code>NAME                   CPU(cores)


myapp-svc-pod1            31m
myapp-svc-pod2            87m
myapp-svc-pod3            1061m
myapp-svc-pod4            35m
myapp-svc-pod5            523m
myapp-svc-pod6            1483m
myapp-svc-pod7            122m
myapp-svc-pod8            562m
</code></pre>

<p>HPA main goal is to spawn more pods to keep average load for a group of pods on specified level.</p>
<p>HPA is not responsible for Load Balancing and equal connection distribution.</p>
<p>For equal connection distribution is responsible k8s service, which works by deafult in iptables mode and - <a href="https://kubernetes.io/docs/concepts/services-networking/service/#proxy-mode-iptables" rel="nofollow noreferrer">according to k8s docs</a> - it picks pods by random.</p>
<p>Your uneven cpu load distribution is most probably caused by the data it processes. To make sure it's not the issue with k8s service, I'd recomment you to export some metrics like number of connections and time it takes to process one request. After you gathered this data, have a look at it and see if a pattern emerges.</p>
<p>Now to answer your question:</p>
<blockquote>
<p>Is this expected and normal?</p>
</blockquote>
<p>It depends what you consider as normal, but if you were expecting more equal cpu load distribution then you may want to rethink your design. It's hard to say what you can do to make it more equal because I don't know what myapp-svc-pods do, but as I already mentioned, it may be best to have a look at the metrics.</p>
