<p>[ Disclaimer: this question was originally posted <a href="https://serverfault.com/questions/939527/kubernetes-calico-on-oracle-cloud-oci">on ServerFault</a>. However, since the official K8s <a href="https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/" rel="nofollow noreferrer">documentation</a> states "ask your questions on StackOverflow", I am also adding it here ]</p>

<p>I am trying to deploy a test Kubernetes cluster on Oracle Cloud, using OCI VM instances - however, I'm having issues with pod networking.</p>

<p>The networking plugin is Calico - it seems to be installed properly, but no traffic gets across the tunnels from one host to another. For example, here I am trying to access nginx running on another node:</p>

<pre><code>root@kube-01-01:~# kubectl get pod -o wide
NAME                    READY   STATUS    RESTARTS   AGE   IP              NODE              NOMINATED NODE
nginx-dbddb74b8-th9ns   1/1     Running   0          38s   192.168.181.1   kube-01-06   &lt;none&gt;
root@kube-01-01:~# curl 192.168.181.1
[ ... timeout... ]
</code></pre>

<p>Using tcpdump, I see the IP-in-IP (protocol 4) packets leaving the first host, but they never seem to make it to the second one (although all other packets, including BGP traffic, make it through just fine).</p>

<pre><code>root@kube-01-01:~# tcpdump -i ens3 proto 4 &amp;
[1] 16642
root@kube-01-01:~# tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes

root@kube-01-01:~# curl 192.168.181.1
09:31:56.262451 IP kube-01-01 &gt; kube-01-06: IP 192.168.21.64.52268 &gt; 192.168.181.1.http: Flags [S], seq 3982790418, win 28000, options [mss 1400,sackOK,TS val 9661065 ecr 0,nop,wscale 7], length 0 (ipip-proto-4)
09:31:57.259756 IP kube-01-01 &gt; kube-01-06: IP 192.168.21.64.52268 &gt; 192.168.181.1.http: Flags [S], seq 3982790418, win 28000, options [mss 1400,sackOK,TS val 9661315 ecr 0,nop,wscale 7], length 0 (ipip-proto-4)
09:31:59.263752 IP kube-01-01 &gt; kube-01-06: IP 192.168.21.64.52268 &gt; 192.168.181.1.http: Flags [S], seq 3982790418, win 28000, options [mss 1400,sackOK,TS val 9661816 ecr 0,nop,wscale 7], length 0 (ipip-proto-4)

root@kube-01-06:~# tcpdump -i ens3 proto 4 &amp;
[1] 12773
root@kube-01-06:~# tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
</code></pre>

<p>What I have checked so far:</p>

<ol>
<li>The Calico routing mesh comes up just fine. I can see the BGP traffic on the packet capture, and I can see all nodes as "up" using calicoctl</li>
</ol>

<p>root@kube-01-01:~# ./calicoctl node status
Calico process is running.</p>

<pre><code>IPv4 BGP status
+--------------+-------------------+-------+----------+-------------+
| PEER ADDRESS |     PEER TYPE     | STATE |  SINCE   |    INFO     |
+--------------+-------------------+-------+----------+-------------+
| 10.13.23.123 | node-to-node mesh | up    | 09:12:50 | Established |
| 10.13.23.124 | node-to-node mesh | up    | 09:12:49 | Established |
| 10.13.23.126 | node-to-node mesh | up    | 09:12:50 | Established |
| 10.13.23.129 | node-to-node mesh | up    | 09:12:50 | Established |
| 10.13.23.127 | node-to-node mesh | up    | 09:12:50 | Established |
| 10.13.23.128 | node-to-node mesh | up    | 09:12:50 | Established |
| 10.13.23.130 | node-to-node mesh | up    | 09:12:52 | Established |
+--------------+-------------------+-------+----------+-------------+
</code></pre>

<ol start="2">
<li>The security rules for the subnet allow all traffic. All the nodes are in the same subnet, and I have a stateless rule permitting all traffic from other nodes within the subnet (I have also tried adding a rule permitting IP-in-IP traffic explicitly - same result).</li>
<li>The source/destination check is disabled on all the vNICs on the K8s nodes.</li>
</ol>

<p>Other things I have noticed:</p>

<ol>
<li>I can get calico to work if I disable IP in IP encapsulation for same-subnet traffic, and use regular routing inside the subnet (as described <a href="https://docs.projectcalico.org/v3.2/reference/public-cloud/aws" rel="nofollow noreferrer">here</a> for AWS)</li>
<li>Other networking plugins (such as weave) seem to work correctly.</li>
</ol>

<p>So my question here is - what is happening to the IP-in-IP encapsulated traffic? Is there anything else I can check to figure out what is going on? </p>

<p>And yes, I know that I could have used managed Kubernetes engine directly, but where is the fun (and the learning opportunity) in that? :D </p>

<p>Edited to address Rico's answer below:</p>

<p>1) I'm also not getting any pod-to-pod traffic to flow through (no communication between pods on different hosts). But I was unable to capture that traffic, so I used node-to-pod as an example.</p>

<p>2) I'm also getting a similar result if I hit a NodePort svc on another node than the one the pod is running on - I see the outgoing IP-in-IP packets from the first node, but they never show up on the second node (the one actually running the pod):</p>

<pre><code>root@kube-01-01:~# tcpdump -i ens3 proto 4 &amp;
[1] 6499
root@kube-01-01:~# tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
root@kube-01-01:~# curl 127.0.0.1:32137
20:24:08.460069 IP kube-01-01 &gt; kube-01-06: IP 192.168.21.64.40866 &gt; 192.168.181.1.http: Flags [S], seq 3175451438, win 43690, options [mss 65495,sackOK,TS val 19444115 ecr 0,nop,wscale 7], length 0 (ipip-proto-4)
20:24:09.459768 IP kube-01-01 &gt; kube-01-06: IP 192.168.21.64.40866 &gt; 192.168.181.1.http: Flags [S], seq 3175451438, win 43690, options [mss 65495,sackOK,TS val 19444365 ecr 0,nop,wscale 7], length 0 (ipip-proto-4)
20:24:11.463750 IP kube-01-01 &gt; kube-01-06: IP 192.168.21.64.40866 &gt; 192.168.181.1.http: Flags [S], seq 3175451438, win 43690, options [mss 65495,sackOK,TS val 19444866 ecr 0,nop,wscale 7], length 0 (ipip-proto-4)
20:24:15.471769 IP kube-01-01 &gt; kube-01-06: IP 192.168.21.64.40866 &gt; 192.168.181.1.http: Flags [S], seq 3175451438, win 43690, options [mss 65495,sackOK,TS val 19445868 ecr 0,nop,wscale 7], length 0 (ipip-proto-4)
</code></pre>

<p>Nothing on the second node ( <code>kube-01-06</code>, the one that is actually running the nginx pod ):</p>

<pre><code>root@kubespray-01-06:~# tcpdump -i ens3 proto 4
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
</code></pre>

<p>I used 127.0.0.1 for ease of demonstration - of course, the exact same thing happens when I hit that NodePort from an outside host:</p>

<pre><code>20:25:17.653417 IP kube-01-01 &gt; kube-01-06: IP 192.168.21.64.56630 &gt; 192.168.181.1.http: Flags [S], seq 980178400, win 64240, options [mss 1440,nop,wscale 8,nop,nop,sackOK], length 0 (ipip-proto-4)
20:25:17.654371 IP kube-01-01 &gt; kube-01-06: IP 192.168.21.64.56631 &gt; 192.168.181.1.http: Flags [S], seq 3932412963, win 64240, options [mss 1440,nop,wscale 8,nop,nop,sackOK], length 0 (ipip-proto-4)
20:25:17.667227 IP kube-01-01 &gt; kube-01-06: IP 192.168.21.64.56632 &gt; 192.168.181.1.http: Flags [S], seq 2017119223, win 64240, options [mss 1440,nop,wscale 8,nop,nop,sackOK], length 0 (ipip-proto-4)
20:25:20.653656 IP kube-01-01 &gt; kube-01-06: IP 192.168.21.64.56630 &gt; 192.168.181.1.http: Flags [S], seq 980178400, win 64240, options [mss 1440,nop,wscale 8,nop,nop,sackOK], length 0 (ipip-proto-4)
20:25:20.654577 IP kube-01-01 &gt; kube-01-06: IP 192.168.21.64.56631 &gt; 192.168.181.1.http: Flags [S], seq 3932412963, win 64240, options [mss 1440,nop,wscale 8,nop,nop,sackOK], length 0 (ipip-proto-4)
20:25:20.668595 IP kube-01-01 &gt; kube-01-06: IP 192.168.21.64.56632 &gt; 192.168.181.1.http: Flags [S], seq 2017119223, win 64240, options [mss 1440,nop,wscale 8,nop,nop,sackOK], length 0 (ipip-proto-4)
</code></pre>

<p>3)  As far as I can tell (please correct me if I'm wrong here), the nodes <em>are</em> aware of routes to pod networks, and pod-to-node traffic is also encapsulated IP-in-IP (notice the protocol 4 packets in the first capture above)</p>

<pre><code>root@kube-01-01:~# kubectl get pod -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP               NODE              NOMINATED NODE
alpine-9d85bf65c-2wx74   1/1     Running   1          23m   192.168.82.194   kube-01-08   &lt;none&gt;
nginx-dbddb74b8-th9ns    1/1     Running   0          10h   192.168.181.1    kube-01-06   &lt;none&gt;

root@kube-01-01:~# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
&lt;snip&gt;
192.168.181.0   10.13.23.127    255.255.255.192 UG    0      0        0 tunl0
</code></pre>

<p>Maybe it is a MTU issue:</p>

<blockquote>
  <p>Typically the MTU for your workload interfaces should match the network MTU. If you need IP-in-IP then the MTU size for both the workload and tunnel interfaces should be 20 bytes less than the network MTU for your network. This is due to the extra 20 byte header that the tunnel will add to each packet.</p>
</blockquote>

<p>Read more <a href="https://docs.projectcalico.org/v3.2/usage/configuration/mtu" rel="nofollow noreferrer">here</a>.</p>
