<p>[ Disclaimer: this question was originally posted <a href="https://serverfault.com/questions/939527/kubernetes-calico-on-oracle-cloud-oci">on ServerFault</a>. However, since the official K8s <a href="https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/" rel="nofollow noreferrer">documentation</a> states "ask your questions on StackOverflow", I am also adding it here ]</p>

<p>I am trying to deploy a test Kubernetes cluster on Oracle Cloud, using OCI VM instances - however, I'm having issues with pod networking.</p>

<p>The networking plugin is Calico - it seems to be installed properly, but no traffic gets across the tunnels from one host to another. For example, here I am trying to access nginx running on another node:</p>

<pre><code>root@kube-01-01:~# kubectl get pod -o wide
NAME                    READY   STATUS    RESTARTS   AGE   IP              NODE              NOMINATED NODE
nginx-dbddb74b8-th9ns   1/1     Running   0          38s   192.168.181.1   kube-01-06   &lt;none&gt;
root@kube-01-01:~# curl 192.168.181.1
[ ... timeout... ]
</code></pre>

<p>Using tcpdump, I see the IP-in-IP (protocol 4) packets leaving the first host, but they never seem to make it to the second one (although all other packets, including BGP traffic, make it through just fine).</p>

<pre><code>root@kube-01-01:~# tcpdump -i ens3 proto 4 &amp;
[1] 16642
root@kube-01-01:~# tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes

root@kube-01-01:~# curl 192.168.181.1
09:31:56.262451 IP kube-01-01 &gt; kube-01-06: IP 192.168.21.64.52268 &gt; 192.168.181.1.http: Flags [S], seq 3982790418, win 28000, options [mss 1400,sackOK,TS val 9661065 ecr 0,nop,wscale 7], length 0 (ipip-proto-4)
09:31:57.259756 IP kube-01-01 &gt; kube-01-06: IP 192.168.21.64.52268 &gt; 192.168.181.1.http: Flags [S], seq 3982790418, win 28000, options [mss 1400,sackOK,TS val 9661315 ecr 0,nop,wscale 7], length 0 (ipip-proto-4)
09:31:59.263752 IP kube-01-01 &gt; kube-01-06: IP 192.168.21.64.52268 &gt; 192.168.181.1.http: Flags [S], seq 3982790418, win 28000, options [mss 1400,sackOK,TS val 9661816 ecr 0,nop,wscale 7], length 0 (ipip-proto-4)

root@kube-01-06:~# tcpdump -i ens3 proto 4 &amp;
[1] 12773
root@kube-01-06:~# tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
</code></pre>

<p>What I have checked so far:</p>

<ol>
<li>The Calico routing mesh comes up just fine. I can see the BGP traffic on the packet capture, and I can see all nodes as "up" using calicoctl</li>
</ol>

<p>root@kube-01-01:~# ./calicoctl node status
Calico process is running.</p>

<pre><code>IPv4 BGP status
+--------------+-------------------+-------+----------+-------------+
| PEER ADDRESS |     PEER TYPE     | STATE |  SINCE   |    INFO     |
+--------------+-------------------+-------+----------+-------------+
| 10.13.23.123 | node-to-node mesh | up    | 09:12:50 | Established |
| 10.13.23.124 | node-to-node mesh | up    | 09:12:49 | Established |
| 10.13.23.126 | node-to-node mesh | up    | 09:12:50 | Established |
| 10.13.23.129 | node-to-node mesh | up    | 09:12:50 | Established |
| 10.13.23.127 | node-to-node mesh | up    | 09:12:50 | Established |
| 10.13.23.128 | node-to-node mesh | up    | 09:12:50 | Established |
| 10.13.23.130 | node-to-node mesh | up    | 09:12:52 | Established |
+--------------+-------------------+-------+----------+-------------+
</code></pre>

<ol start="2">
<li>The security rules for the subnet allow all traffic. All the nodes are in the same subnet, and I have a stateless rule permitting all traffic from other nodes within the subnet (I have also tried adding a rule permitting IP-in-IP traffic explicitly - same result).</li>
<li>The source/destination check is disabled on all the vNICs on the K8s nodes.</li>
</ol>

<p>Other things I have noticed:</p>

<ol>
<li>I can get calico to work if I disable IP in IP encapsulation for same-subnet traffic, and use regular routing inside the subnet (as described <a href="https://docs.projectcalico.org/v3.2/reference/public-cloud/aws" rel="nofollow noreferrer">here</a> for AWS)</li>
<li>Other networking plugins (such as weave) seem to work correctly.</li>
</ol>

<p>So my question here is - what is happening to the IP-in-IP encapsulated traffic? Is there anything else I can check to figure out what is going on? </p>

<p>And yes, I know that I could have used managed Kubernetes engine directly, but where is the fun (and the learning opportunity) in that? :D </p>

<p>Edited to address Rico's answer below:</p>

<p>1) I'm also not getting any pod-to-pod traffic to flow through (no communication between pods on different hosts). But I was unable to capture that traffic, so I used node-to-pod as an example.</p>

<p>2) I'm also getting a similar result if I hit a NodePort svc on another node than the one the pod is running on - I see the outgoing IP-in-IP packets from the first node, but they never show up on the second node (the one actually running the pod):</p>

<pre><code>root@kube-01-01:~# tcpdump -i ens3 proto 4 &amp;
[1] 6499
root@kube-01-01:~# tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
root@kube-01-01:~# curl 127.0.0.1:32137
20:24:08.460069 IP kube-01-01 &gt; kube-01-06: IP 192.168.21.64.40866 &gt; 192.168.181.1.http: Flags [S], seq 3175451438, win 43690, options [mss 65495,sackOK,TS val 19444115 ecr 0,nop,wscale 7], length 0 (ipip-proto-4)
20:24:09.459768 IP kube-01-01 &gt; kube-01-06: IP 192.168.21.64.40866 &gt; 192.168.181.1.http: Flags [S], seq 3175451438, win 43690, options [mss 65495,sackOK,TS val 19444365 ecr 0,nop,wscale 7], length 0 (ipip-proto-4)
20:24:11.463750 IP kube-01-01 &gt; kube-01-06: IP 192.168.21.64.40866 &gt; 192.168.181.1.http: Flags [S], seq 3175451438, win 43690, options [mss 65495,sackOK,TS val 19444866 ecr 0,nop,wscale 7], length 0 (ipip-proto-4)
20:24:15.471769 IP kube-01-01 &gt; kube-01-06: IP 192.168.21.64.40866 &gt; 192.168.181.1.http: Flags [S], seq 3175451438, win 43690, options [mss 65495,sackOK,TS val 19445868 ecr 0,nop,wscale 7], length 0 (ipip-proto-4)
</code></pre>

<p>Nothing on the second node ( <code>kube-01-06</code>, the one that is actually running the nginx pod ):</p>

<pre><code>root@kubespray-01-06:~# tcpdump -i ens3 proto 4
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
</code></pre>

<p>I used 127.0.0.1 for ease of demonstration - of course, the exact same thing happens when I hit that NodePort from an outside host:</p>

<pre><code>20:25:17.653417 IP kube-01-01 &gt; kube-01-06: IP 192.168.21.64.56630 &gt; 192.168.181.1.http: Flags [S], seq 980178400, win 64240, options [mss 1440,nop,wscale 8,nop,nop,sackOK], length 0 (ipip-proto-4)
20:25:17.654371 IP kube-01-01 &gt; kube-01-06: IP 192.168.21.64.56631 &gt; 192.168.181.1.http: Flags [S], seq 3932412963, win 64240, options [mss 1440,nop,wscale 8,nop,nop,sackOK], length 0 (ipip-proto-4)
20:25:17.667227 IP kube-01-01 &gt; kube-01-06: IP 192.168.21.64.56632 &gt; 192.168.181.1.http: Flags [S], seq 2017119223, win 64240, options [mss 1440,nop,wscale 8,nop,nop,sackOK], length 0 (ipip-proto-4)
20:25:20.653656 IP kube-01-01 &gt; kube-01-06: IP 192.168.21.64.56630 &gt; 192.168.181.1.http: Flags [S], seq 980178400, win 64240, options [mss 1440,nop,wscale 8,nop,nop,sackOK], length 0 (ipip-proto-4)
20:25:20.654577 IP kube-01-01 &gt; kube-01-06: IP 192.168.21.64.56631 &gt; 192.168.181.1.http: Flags [S], seq 3932412963, win 64240, options [mss 1440,nop,wscale 8,nop,nop,sackOK], length 0 (ipip-proto-4)
20:25:20.668595 IP kube-01-01 &gt; kube-01-06: IP 192.168.21.64.56632 &gt; 192.168.181.1.http: Flags [S], seq 2017119223, win 64240, options [mss 1440,nop,wscale 8,nop,nop,sackOK], length 0 (ipip-proto-4)
</code></pre>

<p>3)  As far as I can tell (please correct me if I'm wrong here), the nodes <em>are</em> aware of routes to pod networks, and pod-to-node traffic is also encapsulated IP-in-IP (notice the protocol 4 packets in the first capture above)</p>

<pre><code>root@kube-01-01:~# kubectl get pod -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP               NODE              NOMINATED NODE
alpine-9d85bf65c-2wx74   1/1     Running   1          23m   192.168.82.194   kube-01-08   &lt;none&gt;
nginx-dbddb74b8-th9ns    1/1     Running   0          10h   192.168.181.1    kube-01-06   &lt;none&gt;

root@kube-01-01:~# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
&lt;snip&gt;
192.168.181.0   10.13.23.127    255.255.255.192 UG    0      0        0 tunl0
</code></pre>

<p>After a long time and a lot of testing, my belief is that this was caused by IP-in-IP (ipip, or IP protocol 4) traffic being blocked by the Oracle cloud networking layer.</p>
<p>Even though I was unable to find this documented anywhere, it is something that is common for cloud providers (Azure, for example, does the same thing - disallows IP-in-IP and unknown IP traffic).</p>
<p>So the possible workarounds here should be the same ones as the ones listed in the <a href="https://docs.projectcalico.org/reference/public-cloud/azure" rel="nofollow noreferrer">Calico documentation for Azure</a>:</p>
<ul>
<li>Disabling IP-in-IP for same-subnet traffic (as I mentioned in the question)</li>
<li>Switching Calico to VXLAN encapsulation</li>
<li>Using Calico for policy only, and flannel for encapsulation (VXLAN)</li>
</ul>
