<p>I currently have a cluster with 2 nodes:</p>
<ul>
<li>node1 - 192.168.1.20 (master, worker)</li>
<li>node2 - 192.168.1.30 (worker)</li>
</ul>
<p>When running a pod on either of the nodes there is no outbound internet connection.</p>
<p>The problem seems to not be related to DNS since even pinging a public IP address of google.com does not work.</p>
<pre><code>pod1-node1$  traceroute to google.com (216.58.215.46), 30 hops max, 46 byte packets
 1  192-168-1-20.kubernetes.default.svc.home-cloud.local (192.168.1.20)  0.008 ms  0.009 ms  0.009 ms
 2  *
</code></pre>
<p>Nodes have internet access and everything works there</p>
<pre><code>node1$ ping 216.58.215.46 -&gt; OK
node2$ ping 216.58.215.46 -&gt; OK
</code></pre>
<p>Ip tables on master node:</p>
<pre><code>-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N DOCKER
-N DOCKER-INGRESS
-N KUBE-FIREWALL
-N KUBE-KUBELET-CANARY
-N KUBE-LOAD-BALANCER
-N KUBE-MARK-DROP
-N KUBE-MARK-MASQ
-N KUBE-NODE-PORT
-N KUBE-POSTROUTING
-N KUBE-SERVICES
-A PREROUTING -m comment --comment &quot;kubernetes service portals&quot; -j KUBE-SERVICES
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER-INGRESS
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -m comment --comment &quot;kubernetes service portals&quot; -j KUBE-SERVICES
-A OUTPUT -m addrtype --dst-type LOCAL -j DOCKER-INGRESS
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -m comment --comment &quot;kubernetes postrouting rules&quot; -j KUBE-POSTROUTING
-A POSTROUTING -o docker_gwbridge -m addrtype --src-type LOCAL -j MASQUERADE
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.18.0.0/16 ! -o docker_gwbridge -j MASQUERADE
-A POSTROUTING -s 172.17.0.2/32 -d 172.17.0.2/32 -p tcp -m tcp --dport 9000 -j MASQUERADE
-A POSTROUTING -s 172.17.0.3/32 -d 172.17.0.3/32 -p udp -m udp --dport 34197 -j MASQUERADE
-A POSTROUTING -s 172.17.0.3/32 -d 172.17.0.3/32 -p tcp -m tcp --dport 27015 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A DOCKER -i docker_gwbridge -j RETURN
-A DOCKER ! -i docker0 -p tcp -m tcp --dport 9000 -j DNAT --to-destination 172.17.0.2:9000
-A DOCKER ! -i docker0 -p udp -m udp --dport 34197 -j DNAT --to-destination 172.17.0.3:34197
-A DOCKER ! -i docker0 -p tcp -m tcp --dport 27015 -j DNAT --to-destination 172.17.0.3:27015
-A DOCKER-INGRESS -p tcp -m tcp --dport 3306 -j DNAT --to-destination 172.18.0.2:3306
-A DOCKER-INGRESS -p tcp -m tcp --dport 9980 -j DNAT --to-destination 172.18.0.2:9980
-A DOCKER-INGRESS -j RETURN
-A KUBE-FIREWALL -j KUBE-MARK-DROP
-A KUBE-LOAD-BALANCER -j KUBE-MARK-MASQ
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-NODE-PORT -p tcp -m comment --comment &quot;Kubernetes nodeport TCP port for masquerade purpose&quot; -m set --match-set KUBE-NODE-PORT-TCP dst -j KUBE-MARK-MASQ
-A KUBE-POSTROUTING -m comment --comment &quot;Kubernetes endpoints dst ip:port, source ip for solving hairpin purpose&quot; -m set --match-set KUBE-LOOP-BACK dst,dst,src -j MASQUERADE
-A KUBE-POSTROUTING -m mark ! --mark 0x4000/0x4000 -j RETURN
-A KUBE-POSTROUTING -j MARK --set-xmark 0x4000/0x0
-A KUBE-POSTROUTING -m comment --comment &quot;kubernetes service traffic requiring SNAT&quot; -j MASQUERADE --random-fully
-A KUBE-SERVICES -m comment --comment &quot;Kubernetes service lb portal&quot; -m set --match-set KUBE-LOAD-BALANCER dst,dst -j KUBE-LOAD-BALANCER
-A KUBE-SERVICES ! -s 10.233.64.0/18 -m comment --comment &quot;Kubernetes service cluster ip + port for masquerade purpose&quot; -m set --match-set KUBE-CLUSTER-IP dst,dst -j KUBE-MARK-MASQ
-A KUBE-SERVICES -m addrtype --dst-type LOCAL -j KUBE-NODE-PORT
-A KUBE-SERVICES -m set --match-set KUBE-CLUSTER-IP dst,dst -j ACCEPT
-A KUBE-SERVICES -m set --match-set KUBE-LOAD-BALANCER dst,dst -j ACCEPT
</code></pre>
<p>From within a pod I get a strange looking ip route result. My docker overlay network is '10.233.64.0/18' configured with calico.</p>
<pre><code>pod1-node1$ ip route
default via 169.254.1.1 dev eth0 
169.254.1.1 dev eth0 scope link
</code></pre>
<p>I also have metallb installed and it installs a kube-proxy but not sure how exactly it works or if it can be related to the problem.</p>
<p>Any advice is greatly appreciated.
Cheers Alex</p>
<p>EDIT: Cluster is bare metal, installed with kubespray, I haven't installed any CNI manually</p>

sashok_bg
<p>It turned out to be a bug in the CNI - Calico.</p>
<p>I manually set the calico version in my kubespray to 3.22.1 and the problem was solved.</p>
<p>I then encountered other connectivity issues between my nodes and opted to use flannel.
Since then everything works fine.</p>
<p>My advise to others after spending a week of research myself :</p>
<ul>
<li>Container Networking is HARD</li>
</ul>
<p>Take it slow and do not cut corners ! If you are missing some
knowledge about a topic - stop and learn at least the basics of the
said topic before continuing.</p>
<ul>
<li>Diagnose your problem - most commonly it is DNS or Packet Routing:</li>
</ul>
<p>Get the IP of google.com on another machine and ping the IP from within a pod - if it works you have a DNS problem if it doesn't then probably CNI related (packet routing).</p>
<ul>
<li>Learn how resolved works and how it is configured</li>
</ul>
<p>This will allow you to check what DNS is actually used on your node and within your pod.</p>
<ul>
<li>Learn how kubernetes DNS works and why it is needed:</li>
</ul>
<p>It allows pod to service communication to work among other things. It also needs upstream DNS addresses so that it can resolve things outside your cluster.</p>
<ul>
<li>Learn how IP tables work and how masquerading works</li>
</ul>
<p>It allows you to understand how network packets are routed between your node and your pods, using the bridge interface.</p>
<ul>
<li>Learn what a bridge interface is</li>
</ul>
<p>The docker0 bridge interface allows all your containers (and pods by extension) to talk to outside world.</p>
<ul>
<li>Read this very informational article on how calico interacts with your IP tables to create nat masquerading rules on the fly:</li>
</ul>
<p><a href="https://medium.com/@bikramgupta/pod-network-troubleshooting-while-using-calico-on-kubernetes-ee78b731d4d8" rel="nofollow noreferrer">https://medium.com/@bikramgupta/pod-network-troubleshooting-while-using-calico-on-kubernetes-ee78b731d4d8</a></p>
