<p>I've seen scenarios where requests from one workload, sent to a ClusterIP service for another workload with no affinities set, only get routed to a subset of the associated pods. The Endpoints object for this service does show all of the pod IPs.</p>
<p>I did a little experiment to figure out what is happening.</p>
<h2>Experiment</h2>
<p>I set up minikube to have a &quot;router&quot; workload with 3 replicas sending requests to a &quot;backend&quot; workload also with 3 pods. The router just sends a request to the service name like <code>http://backend</code>.</p>
<p>I sent 100 requests to the router service via <code>http://$MINIKUBE_IP:$NODE_PORT</code>, since it's exposed as a <code>NodePort</code> service. Then I observed which backend pods actually handled requests. I repeated this test multiple times.</p>
<p>In most cases, only 2 backend pods handled any requests, with the occasional case where all 3 did. I didn't see any where all requests went to one in these experiments, though I have seen it happen before running other tests in AKS.</p>
<p>This led me to the theory that the router is keeping a persistent connection to the backend pod it connects to. Given there are 3 routers and 3 backends, there's an 11% chance all 3 routers &quot;stick&quot; to a single backend, a 67% chance that between the 3 routers, they stick to 2 of the backends, and a 22% chance that each router sticks to a different backend pod (1-to-1).</p>
<p>Here's one possible combination of router-to-backend connections (out of 27 possible):
<a href="https://i.stack.imgur.com/RR7sj.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/RR7sj.png" alt="three routers sticking to 2 backends" /></a></p>
<h2>Disabling HTTP Keep-Alive</h2>
<p>If I use a <code>Transport</code> disabling HTTP Keep-Alives in the router's http client, then any requests I make to the router are uniformly distributed between the different backends on every test run as desired.</p>
<pre class="lang-golang prettyprint-override"><code>client := http.Client{
    Transport: &amp;http.Transport{
        DisableKeepAlives: true,
    },
}
resp, err := client.Get(&quot;http://backend&quot;)
</code></pre>
<p>So the theory <em>seems</em> accurate. But here's my question:</p>
<ul>
<li>How does the router using HTTP KeepAlive / persistent connections actually result in a single connection between one router pod and one backend pod?
<ul>
<li>There is a <code>kube-proxy</code> in the middle, so I'd expect any persistent connections to be between the router pod and <code>kube-proxy</code> as well as between <code>kube-proxy</code> and the backend pods.</li>
<li>Also, when the router does a DNS lookup, it's going to find the <strong>Cluster IP</strong> of the backend service every time, so how can it &quot;stick&quot; to a Pod if it doesn't know the Pod IP?</li>
</ul>
</li>
</ul>
<p>Using Kubernetes 1.17.7.</p>

Andrew DiNunzio
<p>This <a href="https://learnk8s.io/kubernetes-long-lived-connections" rel="nofollow noreferrer">excellent article</a> covers your question in detail.<br />
Kubernetes Services indeed do not load balance long-lived TCP connections.</p>
<p>Under the hood Services (in most cases) use <code>iptables</code> to distribute connections between pods. But <code>iptables</code> wasn't designed as a balancer, it's a firewall. It isn't capable to do high-level load balancing.<br />
As a weak substitution <code>iptables</code> can create (or not create) a connection to a certain target with some probability - and thus can be used as L3/L4 balancer. This mechanism is what <code>kube-proxy</code> employs to somewhat imitate load balancing.</p>
<blockquote>
<p><strong>Does iptables use round-robin?</strong></p>
<p>No, <code>iptables</code> is primarily used for firewalls, and it is not designed to do load balancing.<br />
However, you could craft a smart set of rules that could make <code>iptables</code> behave like a load balancer.<br />
And this is precisely what happens in Kubernetes.</p>
<p>If you have three Pods, <code>kube-proxy</code> writes the following rules:</p>
<ul>
<li>select Pod 1 as the destination with a likelihood of 33%. Otherwise, move to the next rule</li>
<li>choose Pod 2 as the destination with a probability of 50%. Otherwise, move to the following rule</li>
<li>select Pod 3 as the destination (no probability)</li>
</ul>
<p><strong>What happens when you use keep-alive with a Kubernetes Service?</strong></p>
<p>Let's imagine that front-end and backend support keep-alive.<br />
You have a single instance of the front-end and three replicas for the backend.<br />
The front-end makes the first request to the backend and opens the TCP connection.<br />
The request reaches the Service, and one of the Pod is selected as the destination.<br />
The backend Pod replies and the front-end receives the response.<br />
But instead of closing the TCP connection, it is kept open for subsequent HTTP requests.<br />
What happens when the front-end issues more requests?<br />
They are sent to the same Pod.</p>
<p><strong>Isn't iptables supposed to distribute the traffic?</strong><br />
It is.<br />
There is a single TCP connection open, and iptables rule were invocated the first time.<br />
One of the three Pods was selected as the destination.<br />
Since all subsequent requests are channelled through the same TCP connection, <code>iptables</code> isn't invoked anymore.</p>
</blockquote>
<p>Also it's not quite correct to say that <em><strong>kube-proxy is in the middle</strong></em>.<br />
It isn't - <code>kube-proxy</code> by itself doesn't manage any traffic.<br />
All that it does - it creates iptables rules.<br />
It's <code>iptables</code> who actually listens, distributes, does DNAT etc.</p>
<p>Similar question <a href="https://stackoverflow.com/questions/64876107/kubernetes-requests-not-balanced">here</a>.</p>
