<p>I run Kafka inside Kubernetes cluster on VMWare with a ControlPlane and one worker node. From the ControlPlane node my client can communicate with Kafka, but from my worker node this ends up in this error</p>
<pre><code>   %3|1638529687.405|FAIL|apollo-prototype-765f4d8bcf-bjpf4#producer-2| [thrd:sasl_plaintext://my-cluster-kafka-bootstrap:9092/bootstrap]: sasl_plaintext://my-cluster-kafka-bootstrap:9092/bootstrap: Failed to resolve 'my-cluster-kafka-bootstrap:9092': Temporary failure in name resolution (after 20016ms in state CONNECT, 2 identical error(s) suppressed)
   %3|1638529687.406|ERROR|apollo-prototype-765f4d8bcf-bjpf4#producer-2| [thrd:app]: apollo-prototype-765f4d8bcf-bjpf4#producer-2: sasl_plaintext://my-cluster-kafka-bootstrap:9092/bootstrap: Failed to resolve 'my-cluster-kafka-bootstrap:9092': Temporary failure in name resolution (after 20016ms in state CONNECT, 2 identical error(s) suppressed)
</code></pre>
<p>This is my Kafka cluster manifest (using Strimzi)</p>
<pre><code>listeners:
  - name: plain
    port: 9092
    type: internal
    tls: false
    authentication:
      type: scram-sha-512
  - name: external
    port: 9094
    type: ingress
    tls: true
    authentication:
      type: scram-sha-512
    configuration:
      class: nginx
      bootstrap:
        host: localb.kafka.xxx.com
      brokers:
      - broker: 0
        host: local.kafka.xxx.com
</code></pre>
<p>To be mentioned that exactly the same config, when i run inside in a cloud works flawleslly.</p>
<p><strong>Telnet</strong> and <strong>nslookup</strong> (from both nodes) throw up an error.
CoreDNS logs do not even mention this error.
Also firewall is disable on both nodes.</p>
<p>Could you please help me out? Thanks!</p>
<hr />
<p><strong>UPDATE: SOLUTION</strong>
Calico Pod (from the worker node) was complaining that <strong>bird: Netlink: Network is down</strong>, even it was not crashing</p>
<pre><code>2021-12-03 09:39:58.051 [INFO][90] felix/int_dataplane.go 1539: Received interface addresses update msg=&amp;intdataplane.ifaceAddrsUpdate{Name:&quot;tunl0&quot;, Addrs:set.mapSet{}}
2021-12-03 09:39:58.051 [INFO][90] felix/hostip_mgr.go 85: Interface addrs changed. update=&amp;intdataplane.ifaceAddrsUpdate{Name:&quot;tunl0&quot;, Addrs:set.mapSet{}}
2021-12-03 09:39:58.052 [INFO][90] felix/ipsets.go 130: Queueing IP set for creation family=&quot;inet&quot; setID=&quot;this-host&quot; setType=&quot;hash:ip&quot;
2021-12-03 09:39:58.057 [INFO][90] felix/ipsets.go 785: Doing full IP set rewrite family=&quot;inet&quot; numMembersInPendingReplace=3 setID=&quot;this-host&quot;
2021-12-03 09:39:58.059 [INFO][90] felix/int_dataplane.go 1036: Linux interface state changed. ifIndex=13 ifaceName=&quot;tunl0&quot; state=&quot;down&quot;
2021-12-03 09:39:58.082 [INFO][90] felix/int_dataplane.go 1521: Received interface update msg=&amp;intdataplane.ifaceUpdate{Name:&quot;tunl0&quot;, State:&quot;down&quot;, Index:13}
bird: Netlink: Network is down
</code></pre>
<p><a href="https://github.com/projectcalico/calico/issues/4273#issuecomment-752302936" rel="nofollow noreferrer">Here</a> is what I have done and it worked like a charm!</p>
<blockquote>
<p>The fault is caused by the different ipvs modules loaded by the node.
I configured the ipip module for the new node, but the old node did
not load the ipip module, which caused the calico exception. Delete
the ipip module to return to normal.</p>
<pre><code>[root@k8s-node236-232 ~]# lsmod  | grep ipip
ipip                   16384  0 
tunnel4                16384  1 ipip
ip_tunnel              24576  1 ipip
[root@k8s-node236-232 ~]# modprobe -r ipip
[root@k8s-node236-232 ~]# lsmod  | grep ipip
</code></pre>
</blockquote>

Oana
<p>Calico Pod (from the worker node) was complaining that <strong>bird: Netlink: Network is down</strong>, even it was not crashing</p>
<pre><code>2021-12-03 09:39:58.051 [INFO][90] felix/int_dataplane.go 1539: Received interface addresses update msg=&amp;intdataplane.ifaceAddrsUpdate{Name:&quot;tunl0&quot;, Addrs:set.mapSet{}}
2021-12-03 09:39:58.051 [INFO][90] felix/hostip_mgr.go 85: Interface addrs changed. update=&amp;intdataplane.ifaceAddrsUpdate{Name:&quot;tunl0&quot;, Addrs:set.mapSet{}}
2021-12-03 09:39:58.052 [INFO][90] felix/ipsets.go 130: Queueing IP set for creation family=&quot;inet&quot; setID=&quot;this-host&quot; setType=&quot;hash:ip&quot;
2021-12-03 09:39:58.057 [INFO][90] felix/ipsets.go 785: Doing full IP set rewrite family=&quot;inet&quot; numMembersInPendingReplace=3 setID=&quot;this-host&quot;
2021-12-03 09:39:58.059 [INFO][90] felix/int_dataplane.go 1036: Linux interface state changed. ifIndex=13 ifaceName=&quot;tunl0&quot; state=&quot;down&quot;
2021-12-03 09:39:58.082 [INFO][90] felix/int_dataplane.go 1521: Received interface update msg=&amp;intdataplane.ifaceUpdate{Name:&quot;tunl0&quot;, State:&quot;down&quot;, Index:13}
bird: Netlink: Network is down
</code></pre>
<p><a href="https://github.com/projectcalico/calico/issues/4273#issuecomment-752302936" rel="nofollow noreferrer">Here</a> is what I have done and it worked like a charm!</p>
<blockquote>
<p>The fault is caused by the different ipvs modules loaded by the node.
I configured the ipip module for the new node, but the old node did
not load the ipip module, which caused the calico exception. Delete
the ipip module to return to normal.</p>
<pre><code>[root@k8s-node236-232 ~]# lsmod  | grep ipip
ipip                   16384  0 
tunnel4                16384  1 ipip
ip_tunnel              24576  1 ipip
[root@k8s-node236-232 ~]# modprobe -r ipip
[root@k8s-node236-232 ~]# lsmod  | grep ipip
</code></pre>
</blockquote>
