<p>I have a k8s cluster with an <code>ipvs</code> kube-proxy mode and a database cluster outside of k8s.</p>
<p>In order to get access to the DB cluster I created service and endpoints resources:</p>
<pre><code>---
apiVersion: v1
kind: Service
metadata:
  name: database
spec:
  type: ClusterIP
  ports:
  - protocol: TCP
    port: 3306
    targetPort: 3306

---
apiVersion: v1
kind: Endpoints
metadata:
  name: database
subsets:
- addresses:
  - ip: 192.168.255.9
  - ip: 192.168.189.76
  ports:
  - port: 3306
    protocol: TCP
</code></pre>
<p>Then I run a pod with MySQL client and try to connect to this service:</p>
<pre><code>mysql -u root -p password -h database
</code></pre>
<p>In the network dump I see a successful TCP handshake and successful MySQL connection. On the node where the pod is running (hereinafter the worker node) I see the next established connection:</p>
<pre><code>sudo netstat-nat -n | grep 3306
tcp   10.0.198.178:52642             192.168.189.76:3306            ESTABLISHED
</code></pre>
<p>Then I send some test queries from the pod in an opened MySQL session. They all are sent to the same node. It's expected behavior.</p>
<p>Then I monitor established connections on the worker node. After about 5 minutes the established connection to the database node is missed.</p>
<p>But in the network dump I see that TCP finalization packets are not sent from the worker node to the database node. As a result, I get a leaked connection on the database node.</p>
<p>How <code>ipvs</code> decides to drop an established connection? If <code>ipvs</code> drops a connection, why it doesn't finalize TCP connection properly? Is it a bug or do I misunderstand something with an <code>ipvs</code> mode in kube-proxy?</p>

Al Ryz
<p>Kube-proxy and Kubernetes don't help to balance persistent connections.</p>
<p>The whole concept of the long-lived connections in Kubernetes is well described in <a href="https://learnk8s.io/kubernetes-long-lived-connections" rel="nofollow noreferrer">this article</a>:</p>
<blockquote>
<p>Kubernetes doesn't load balance long-lived connections, and some Pods
might receive more requests than others. If you're using HTTP/2, gRPC,
RSockets, AMQP or any other long-lived connection such as a database
connection, you might want to consider client-side load balancing.</p>
</blockquote>
<p>I recommend going through the whole thing but overall it can be summed up with:</p>
<blockquote>
<ul>
<li><p>Kubernetes Services are designed to cover most common uses for web applications.</p>
</li>
<li><p>However, as soon as you start working with application protocols that use persistent TCP connections, such as databases, gRPC, or
WebSockets, they fall apart.</p>
</li>
<li><p>Kubernetes doesn't offer any built-in mechanism to load balance long-lived TCP connections.</p>
</li>
<li><p>Instead, you should code your application so that it can retrieve and load balance upstreams client-side.</p>
</li>
</ul>
</blockquote>
