<p>I know the below information is not enough to trace the issue but still, I want some solution.</p>
<p>We have Amazon EKS cluster.</p>
<p>Currently, we are facing the reachability of the Kafka pod issue.</p>
<p><strong>Environment:</strong></p>
<ul>
<li>Total 10 nodes with Availability zone ap-south-1a,1b</li>
<li>I have a three replica of the Kafka cluster (Helm chart installation)</li>
<li>I have a three replica of the zookeeper (Helm chart installation)</li>
<li>Kafka using external advertised listener on port 19092</li>
<li>Kafka has service with an internal network load balancer</li>
<li>I have deployed a test-pod to check reachability of Kafka pod.</li>
<li>we are using cloud-map based DNS for advertized listener</li>
</ul>
<p><strong>Working:</strong></p>
<ul>
<li>When I run telnet command from ec2 like <code>telnet 10.0.1.45 19092</code>. It works as expected. IP <code>10.0.1.45</code> is a loadbalancer ip.</li>
<li>When I run telnet command from ec2 like <code>telnet 10.0.1.69 31899</code>. It works as expected. IP <code>10.0.1.69</code> is a actual node's ip and 31899 is nodeport.</li>
</ul>
<p><strong>Problem:</strong></p>
<ul>
<li>When I run same command from test-pod. like <code>telnet 10.0.1.45 19092</code>. It works sometime and sometime it will gives an error like <code>telnet: Unable to connect to remote host: Connection timed out</code></li>
</ul>
<p>The issue is something related to kube-proxy. we need help to resolve this issue.</p>
<p>Can anyone help to guide me?
Can I restart kube-proxy? Does it affect other pods/deployments?</p>

<p>I believe this problem is caused by AWS's NLB TCP-only nature (as mentioned in the comments).</p>
<p>In a nutshell, <a href="https://github.com/kubernetes/kubernetes/issues/66796#issuecomment-409066184" rel="nofollow noreferrer">your pod-to-pod communication fails when hairpin is needed</a>.</p>
<p>To confirm this is the root cause, you can verify that when the telnet works, kafka pod and client pod <strong>are not</strong> in the same EC2 node. And when they're in the same EC2 server, the telnet fails.</p>
<p>There are (at least) two approaches to tackle this issue:</p>
<ol>
<li><strong>Use K8s internal networking</strong> - Refer to k8s Service's URL</li>
</ol>
<p>Every K8s service has <a href="https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#namespaces-of-services" rel="nofollow noreferrer">its own DNS FQDN</a> for internal usage (meaning using k8s network only, without reaching the LoadBalancer and come back to k8s again). You can just telnet this instead of the NodePort via the LB.
I.e. let's assume your kafka service is named <code>kafka</code>. Then you can just telnet <code>kafka.svc.cluster.local</code> (on the port exposed by kafka service)</p>
<ol start="2">
<li><strong>Use <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#inter-pod-affinity-and-anti-affinity" rel="nofollow noreferrer">K8s anti-affinity</a></strong> to make sure client and kafka are never scheduled in the same node.</li>
</ol>
<p>Oh and as indicated in <a href="https://stackoverflow.com/questions/34937184/kubernetes-cant-connect-to-a-service-ip-from-the-services-pod?rq=1">this answer</a> you might need to make that service <strong>headless</strong>.</p>
