<p>In Kubernetes services talk to each other via a service ip. With iptables or something similar each TCP connection is transparently routed to one of the pods that are available for the called service. If the calling service is not closing the TCP connection (e.g. using TCP keepalive or a connection pool) it will connect to one pod and not use the other pods that may be spawned.</p>

<p>What is the correct way to handle such a situation?</p>

<hr>

<p>My own unsatisfying ideas:</p>

<h2>Closing connection after each api call</h2>

<p>Am I making every call slower only to be able to distribute requests to different pods? Doesn't feel right.</p>

<h2>Minimum number of connections</h2>

<p>I could force the caller to open multiple connections (assuming it would then distribute the requests across these connections) but how many should be open? The caller has (and probably should not have) no idea how many pods there are.</p>

<h2>Disable bursting</h2>

<p>I could limit the resources of the called services so it gets slow on multiple requests and the caller will open more connections (hopefully to other pods). Again I don't like the idea of arbitrarily slowing down the requests and this will only work on cpu bound services.</p>

<p>The keep-alive behavior can be tuned by options specified in the Keep-Alive general header:</p>

<p>E.g:</p>

<pre><code>Connection: Keep-Alive
Keep-Alive: max=10, timeout=60
</code></pre>

<p>Thus, you could re-open a tcp connection after a specific timeout instead than at each API request or after a max number of http transactions.</p>

<p>Keep in mind that timeout and max are not guaranteed.</p>

<p>EDIT:</p>

<p>Note that If you use k8s service you can choose two LB mode:</p>

<ul>
<li><p>iptables proxy mode (By default, kube-proxy in iptables mode chooses a backend at random.)</p></li>
<li><p>IPVS proxy mode where you have different load balancing options:</p></li>
</ul>

<p>IPVS provides more options for balancing traffic to backend Pods; these are:</p>

<p>rr: round-robin
lc: least connection (smallest number of open connections)
dh: destination hashing
sh: source hashing
sed: shortest expected delay
nq: never queue</p>

<p>check <a href="https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies" rel="nofollow noreferrer">this link</a></p>
