<p>I'm probably missing something in my understanding of networking and gRPC. I want to start with an example to explain how I understand the networking.</p>
<p>I have a deployment with 3 replicas named app in default namespace
they have pod IP:</p>
<p><code>10.3.0.1</code>, <code>10.3.0.2</code>, <code>10.3.0.3</code></p>
<p>I have a <code>ClusterIP</code> service for the deployment called app-service, it has an IP of:</p>
<p><code>10.24.0.0</code></p>
<p>The kube-dns will have a record that maps the <code>app-service.default.svc.cluster.local</code> -&gt; <code>10.24.0.0</code>. The kube-proxy on every node will see the configuration and update the netfilter with mapping <code>10.24.0.0</code> -&gt; <code>10.3.0.1</code>, <code>10.3.0.2</code>, <code>10.3.0.3</code>.</p>
<p>Now in my cluster I have another client pod that makes a gRPC call to <code>app-service.default.svc.cluster.local</code>.</p>
<p>What I expect to happen is <code>app-service.default.svc.cluster.local</code> will resolve to a single IP <code>10.24.0.0</code>, and gRPC will create a single sub channel and try to establish a long-live connection.</p>
<p>This call will get out of the pod and to the node and go pass the netfilter at which point <code>10.24.0.0</code> becomes <code>10.3.0.1</code> and finally arrives on the first pod.</p>
<p>Now there is a second call from the client, it goes through the netfilter at which point <code>10.24.0.0</code> becomes <code>10.3.0.2</code>, this call ends up on another pod which had no idea they had a connection established?</p>
<p>Also I see these <a href="https://techdozo.dev/grpc-load-balancing-on-kubernetes-using-headless-service/" rel="nofollow noreferrer">blogs</a> that mention gRPC will create sticky session to a single pod IP, but I thought the pod IP will not be resolved inside the application/grpc client but in the netfilter.</p>

<p>I think you could find some answers from the <a href="https://github.com/grpc/grpc/blob/master/doc/load-balancing.md" rel="nofollow noreferrer">Load Balancing in gRPC</a>.</p>
<p>Load-balancing policies fit into the gRPC client workflow in between name resolution and the connection to the server.</p>
<p><a href="https://i.stack.imgur.com/mLX6s.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/mLX6s.png" alt="enter image description here" /></a></p>
<ul>
<li>On startup, the gRPC client issues a <a href="https://github.com/grpc/grpc/blob/master/doc/naming.md" rel="nofollow noreferrer">name resolution</a> request for the server name. The name will resolve to <strong>a list of IP addresses</strong>, a service config that indicates which client-side load-balancing policy to use (e.g., <code>round_robin</code> or <code>grpclb</code>) and provides a configuration for that policy and a set of attributes (channel args in C-core).</li>
<li>The client instantiates the load balancing policy and passes its configuration from the service config, the list of IP addresses, and the attributes.</li>
<li>The load balancing policy creates a set of subchannels for the IP addresses of the servers (which might be different from the IP addresses returned by the resolver; see below). It also watches the subchannels' connectivity states and decides when each subchannel should attempt to connect.</li>
<li>For each RPC sent, the load balancing policy decides which subchannel (i.e., which server) the RPC should be sent to.</li>
</ul>
<p>When the IP of pods is changed due to pods restart, now gRPC will try to re-resolve whenever a subconn goes into transient failure. Here is one related <a href="https://github.com/grpc/grpc-go/issues/1663" rel="nofollow noreferrer">discussion</a> and <a href="https://github.com/grpc/grpc-go/blob/master/clientconn.go#L1258" rel="nofollow noreferrer">code</a></p>
