<p>I have Kubernets 1.20.1 cluster with single master and single worker configured with <code>ipvs</code> mode. Using calico CNI <code>calico/cni:v3.16.1</code>. Cluster running on OS RHEL 8 kernel <code>4.18.0-240.10</code> with firewalld and selinux disabled.</p>
<p>Running one <code>netshoot</code> pod (<code>10.1.30.130</code>) on master and another pod (<code>10.3.65.132</code>) in worker node.</p>
<ol>
<li>I can ping both pod, in both direction</li>
<li>if run the nc command in web server mode, connection is not working. I tried to run nginx on both server, not able get http traffic one server from another server.</li>
</ol>
<p>Ran the tcpdump on both servers <code>tcpdump -vv -nn -XX -i any host &lt;PODIP&gt;</code> I can see ping traffic going to both nodes, but TCP traffic not reaching the other node.</p>
<p><code>iptables -vL | grep DROP</code> command not showing any packet drop on both nodes.</p>
<p>I don't know where the TCP traffic getting lost, need some tips to troubleshoot this issue.</p>
<p><strong>Master node iptables-save command output</strong></p>
<pre><code># Generated by iptables-save v1.8.4 on Sat Jan 16 18:52:50 2021
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:KUBE-MARK-DROP - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-POSTROUTING - [0:0]
:KUBE-KUBELET-CANARY - [0:0]
:KUBE-SERVICES - [0:0]
:KUBE-FIREWALL - [0:0]
:KUBE-NODE-PORT - [0:0]
:KUBE-LOAD-BALANCER - [0:0]
-A PREROUTING -m comment --comment &quot;kubernetes service portals&quot; -j KUBE-SERVICES
-A POSTROUTING -m comment --comment &quot;kubernetes postrouting rules&quot; -j KUBE-POSTROUTING
-A OUTPUT -m comment --comment &quot;kubernetes service portals&quot; -j KUBE-SERVICES
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-POSTROUTING -m comment --comment &quot;Kubernetes endpoints dst ip:port, source ip for solving hairpin purpose&quot; -m set --match-set KUBE-LOOP-BACK dst,dst,src -j MASQUERADE
-A KUBE-POSTROUTING -m mark ! --mark 0x4000/0x4000 -j RETURN
-A KUBE-POSTROUTING -j MARK --set-xmark 0x4000/0x0
-A KUBE-POSTROUTING -m comment --comment &quot;kubernetes service traffic requiring SNAT&quot; -j MASQUERADE --random-fully
-A KUBE-SERVICES ! -s 10.0.0.0/14 -m comment --comment &quot;Kubernetes service cluster ip + port for masquerade purpose&quot; -m set --match-set KUBE-CLUSTER-IP dst,dst -j KUBE-MARK-MASQ
-A KUBE-SERVICES -m addrtype --dst-type LOCAL -j KUBE-NODE-PORT
-A KUBE-SERVICES -m set --match-set KUBE-CLUSTER-IP dst,dst -j ACCEPT
-A KUBE-FIREWALL -j KUBE-MARK-DROP
-A KUBE-LOAD-BALANCER -j KUBE-MARK-MASQ
COMMIT
# Completed on Sat Jan 16 18:52:50 2021
# Generated by iptables-save v1.8.4 on Sat Jan 16 18:52:50 2021
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:KUBE-FIREWALL - [0:0]
:KUBE-KUBELET-CANARY - [0:0]
:KUBE-FORWARD - [0:0]
-A INPUT -j KUBE-FIREWALL
-A FORWARD -m comment --comment &quot;kubernetes forwarding rules&quot; -j KUBE-FORWARD
-A OUTPUT -j KUBE-FIREWALL
-A KUBE-FIREWALL -m comment --comment &quot;kubernetes firewall for dropping marked packets&quot; -m mark --mark 0x8000/0x8000 -j DROP
-A KUBE-FIREWALL ! -s 127.0.0.0/8 -d 127.0.0.0/8 -m comment --comment &quot;block incoming localnet connections&quot; -m conntrack ! --ctstate RELATED,ESTABLISHED,DNAT -j DROP
-A KUBE-FORWARD -m comment --comment &quot;kubernetes forwarding rules&quot; -m mark --mark 0x4000/0x4000 -j ACCEPT
-A KUBE-FORWARD -m comment --comment &quot;kubernetes forwarding conntrack pod source rule&quot; -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-FORWARD -m comment --comment &quot;kubernetes forwarding conntrack pod destination rule&quot; -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
COMMIT
# Completed on Sat Jan 16 18:52:50 2021
# Generated by iptables-save v1.8.4 on Sat Jan 16 18:52:50 2021
*mangle
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:KUBE-KUBELET-CANARY - [0:0]
COMMIT
# Completed on Sat Jan 16 18:52:50 2021
</code></pre>
<p><strong>Worker iptables-save output</strong></p>
<pre><code># Generated by iptables-save v1.8.4 on Sat Jan 16 18:53:58 2021
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:KUBE-MARK-DROP - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-POSTROUTING - [0:0]
:KUBE-KUBELET-CANARY - [0:0]
:KUBE-SERVICES - [0:0]
:KUBE-FIREWALL - [0:0]
:KUBE-NODE-PORT - [0:0]
:KUBE-LOAD-BALANCER - [0:0]
-A PREROUTING -m comment --comment &quot;kubernetes service portals&quot; -j KUBE-SERVICES
-A POSTROUTING -m comment --comment &quot;kubernetes postrouting rules&quot; -j KUBE-POSTROUTING
-A OUTPUT -m comment --comment &quot;kubernetes service portals&quot; -j KUBE-SERVICES
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-POSTROUTING -m mark ! --mark 0x4000/0x4000 -j RETURN
-A KUBE-POSTROUTING -j MARK --set-xmark 0x4000/0x0
-A KUBE-POSTROUTING -m comment --comment &quot;kubernetes service traffic requiring SNAT&quot; -j MASQUERADE --random-fully
-A KUBE-SERVICES ! -s 10.0.0.0/14 -m comment --comment &quot;Kubernetes service cluster ip + port for masquerade purpose&quot; -m set --match-set KUBE-CLUSTER-IP dst,dst -j KUBE-MARK-MASQ
-A KUBE-SERVICES -m addrtype --dst-type LOCAL -j KUBE-NODE-PORT
-A KUBE-SERVICES -m set --match-set KUBE-CLUSTER-IP dst,dst -j ACCEPT
-A KUBE-FIREWALL -j KUBE-MARK-DROP
-A KUBE-LOAD-BALANCER -j KUBE-MARK-MASQ
COMMIT
# Completed on Sat Jan 16 18:53:58 2021
# Generated by iptables-save v1.8.4 on Sat Jan 16 18:53:58 2021
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:KUBE-FIREWALL - [0:0]
:KUBE-KUBELET-CANARY - [0:0]
:KUBE-FORWARD - [0:0]
-A INPUT -j KUBE-FIREWALL
-A FORWARD -m comment --comment &quot;kubernetes forwarding rules&quot; -j KUBE-FORWARD
-A OUTPUT -j KUBE-FIREWALL
-A KUBE-FIREWALL -m comment --comment &quot;kubernetes firewall for dropping marked packets&quot; -m mark --mark 0x8000/0x8000 -j DROP
-A KUBE-FIREWALL ! -s 127.0.0.0/8 -d 127.0.0.0/8 -m comment --comment &quot;block incoming localnet connections&quot; -m conntrack ! --ctstate RELATED,ESTABLISHED,DNAT -j DROP
-A KUBE-FORWARD -m comment --comment &quot;kubernetes forwarding rules&quot; -m mark --mark 0x4000/0x4000 -j ACCEPT
-A KUBE-FORWARD -m comment --comment &quot;kubernetes forwarding conntrack pod source rule&quot; -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-FORWARD -m comment --comment &quot;kubernetes forwarding conntrack pod destination rule&quot; -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
COMMIT
# Completed on Sat Jan 16 18:53:58 2021
# Generated by iptables-save v1.8.4 on Sat Jan 16 18:53:58 2021
*mangle
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:KUBE-KUBELET-CANARY - [0:0]
COMMIT
# Completed on Sat Jan 16 18:53:58 2021
</code></pre>

<p>I was able to resolve this issue by running below command on <code>ens192</code> interface on VMware VM on.</p>
<pre><code># cat /etc/sysconfig/network-scripts/ifcfg-ens192 | grep ETHTOOL
ETHTOOL_OPTS=&quot;-K ens192 tx-udp_tnl-csum-segmentation off; -K ens192 tx-udp_tnl-segmentation off&quot;
</code></pre>
<p>got the tips from here:
<a href="https://github.com/kubernetes-sigs/kubespray/issues/7268" rel="nofollow noreferrer">https://github.com/kubernetes-sigs/kubespray/issues/7268</a>
Thanks</p>
<p>SR</p>
