<p>I'm starting with K8S. I installed 2 Debian 10 VMs on Azure (1 master node &amp; 2 slaves). </p>

<p>I installed the master node with this doc: 
<a href="https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/" rel="nofollow noreferrer">https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/</a></p>

<p>I installed Calico with this one : 
<a href="https://docs.projectcalico.org/getting-started/kubernetes/installation/calico#installing-with-the-kubernetes-api-datastore50-nodes-or-less" rel="nofollow noreferrer">https://docs.projectcalico.org/getting-started/kubernetes/installation/calico#installing-with-the-kubernetes-api-datastore50-nodes-or-less</a></p>

<p>I created a simple nginx deployment:</p>

<pre><code>kubectl run nginx --replicas=2 --image=nginx
</code></pre>

<p>I have the following pods (sazultk8s1/2 are the working nodes) :</p>

<pre><code>root@itf-infra-sazultk8s0-vm:~# kubectl get pods -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP               NODE                          
nginx-6db489d4b7-mzmnq              1/1     Running   0          12s   192.168.47.18     itf-infra-sazultk8s2-vm   
nginx-6db489d4b7-sgdz7              1/1     Running   0          12s   192.168.247.115   itf-infra-sazultk8s1-vm
</code></pre>

<p>From the master node I can't curl to these nginx: </p>

<pre><code>root@itf-infra-sazultk8s0-vm:~# curl 192.168.47.18 --connect-timeout 5
curl: (28) Connection timed out after 5001 milliseconds
root@itf-infra-sazultk8s0-vm:~# curl 192.168.247.115 --connect-timeout 5
curl: (28) Connection timed out after 5000 milliseconds
</code></pre>

<p>I tried from a simple busybox image: </p>

<pre><code>kubectl run access --rm -ti --image busybox /bin/sh
/ #ifconfig eth0 | grep -i inet
   inet addr:192.168.247.116  Bcast:0.0.0.0  Mask:255.255.255.255
/ # wget --timeout 5 192.168.247.115
Connecting to 192.168.247.115 (192.168.247.115:80)
saving to 'index.html'
index.html           100% |********************************************************************************************************|   612  0:00:00 ETA
'index.html' saved
/ # wget --timeout 5 192.168.47.18
Connecting to 192.168.47.18 (192.168.47.18:80)
wget: download timed out
</code></pre>

<p>From a scratch install: </p>

<ol>
<li>does a pod can ping a pod on another host ? </li>
<li>is it possible to curl from master node to a pod on a worker node ? </li>
<li>does azure apply restrictions and prevent k8s to work properly ? </li>
</ol>

<p>Took me 1 week to solve it.</p>
<ul>
<li>From the master node, you want to ping/curl Pods located on worker nodes. These Pods are part of a Deployment, itself exposed through a Service.</li>
<li>There are some subtilities in Azure networking which make this not &quot;working out of the box&quot; with default Calico installation.</li>
</ul>
<p><strong>Steps to make Calico work on Azure</strong></p>
<ol>
<li>In Kubernetes, Install Calico without a networking backend.</li>
<li>In Azure, Enable IP forwarding on each host.</li>
<li>In Azure, Create UDR (user Defined Routes).</li>
</ol>
<p><strong>1. Kubernetes, Install Calico without a networking backend</strong></p>
<p>A) Disable Bird
By default, calico.yaml is configured to use <code>bird</code> as a network backend, you have to set it to <code>none</code>.
Official installation step: <a href="https://docs.projectcalico.org/getting-started/kubernetes/self-managed-onprem/onpremises" rel="nofollow noreferrer">https://docs.projectcalico.org/getting-started/kubernetes/self-managed-onprem/onpremises</a></p>
<p>Before applying -f <code>calico.yaml</code>, edit the file.
Search for the variable <code>CALICO_NETWORKING_BACKEND</code></p>
<p><a href="https://i.stack.imgur.com/STJoB.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/STJoB.png" alt="enter image description here" /></a></p>
<p>We see that the value is taken from a ConfigMap.
Edit the value in the ConfigMap (located at the top of the file), to set it to <code>none</code> instead of the default <code>bird</code>.</p>
<p><a href="https://i.stack.imgur.com/NkNz9.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/NkNz9.png" alt="enter image description here" /></a></p>
<p>B) Remove Bird from the Readiness &amp; Liveliness probes</p>
<p>Given that we have disabled Bird, it should be removed from the Readiness &amp; Liveliness probes, otherwise, the calico-node deamonset pods won't start. In Calico Manifest, comment out &quot;- -bird-live&quot; and &quot;- bird-ready&quot;.</p>
<p><a href="https://i.stack.imgur.com/booqz.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/booqz.png" alt="enter image description here" /></a></p>
<p>You are done here, you can apply the file: kubectl apply -f</p>
<p><strong>2. Azure, Enable IP forwarding on each host</strong></p>
<p>For each VM in Azure:</p>
<ul>
<li>Click on it &gt; Networking &gt; click on the Network Interface you have.
<a href="https://i.stack.imgur.com/xM7Si.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/xM7Si.png" alt="enter image description here" /></a></li>
<li>Click on IP Configurations</li>
<li>Set IP forwarding to Enabled.
<a href="https://i.stack.imgur.com/Y1ytx.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Y1ytx.png" alt="enter image description here" /></a></li>
</ul>
<p>Repeat for each VM, and you are done.</p>
<p>Note: as per the <a href="https://learn.microsoft.com/en-us/azure/virtual-network/virtual-network-network-interface#enable-or-disable-ip-forwarding" rel="nofollow noreferrer">Azure doc</a>, IP forwarding enables the virtual machine a network interface is attached to:</p>
<ul>
<li>Receive network traffic not destined for one of the IP addresses assigned to any of the IP configurations assigned to the network interface.</li>
<li>Send network traffic with a different source IP address than the one assigned to one of a network interface's IP configurations.</li>
</ul>
<p><strong>3. Azure, Create UDR (User Defined Routes)</strong></p>
<p>Next, you have to create UDR on your Azure subnet, so that Azure can route the traffic targeted to the (Pod subnet created by Calico on the target Host), to the (IP of the actual target Host itself). So that Azure know that the traffic aimed to that calico subnet, has to be routed to the appropriate node, otherwise Azure doesn't know what to do with this traffic.
Then, when the target node is reached, the target knows how to route the traffic to its underlying Pods.</p>
<p>First, identify the subnet created by Calico on each node.</p>
<pre><code>kubectl get ipamblocks.crd.projectcalico.org \
-o jsonpath=&quot;{range .items[*]}{'podNetwork: '}{.spec.cidr}{'\t NodeIP: '}{.spec.affinity}{'\n'}&quot;
</code></pre>
<p><a href="https://i.stack.imgur.com/5noXh.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/5noXh.png" alt="enter image description here" /></a></p>
<p>On Azure, <a href="https://learn.microsoft.com/en-us/azure/virtual-network/tutorial-create-route-table-portal#create-a-route-table" rel="nofollow noreferrer">follows the documentation</a> on how to 'Create a route Table', 'Add Routes of the table', and to 'Associate the route Table to a subnet' (just scroll the doc, sections are one below the other).</p>
<p>The final result should look like this:
<a href="https://i.stack.imgur.com/gD1Y6.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/gD1Y6.png" alt="enter image description here" /></a></p>
<p>You are done! You should now be able to ping/curl your Pods located on other nodes.</p>
<p><strong>References Links</strong></p>
<p>All the reference links expaining the subtilities of Azure Networking, and the different ways to use Calico with Azure (Network+NetworkPolicy, or NetworkPolicy only).</p>
<p>In particular, there are 3 ways to make Calico work on Azure.</p>
<ol>
<li>The one we just see, where the routes are managed by the User. It seems that this could be called &quot;user managed networking&quot;.</li>
<li>Using Azure CNI IPAM plugin. Here we could say &quot;Azure managed networking&quot;. Azure will allocate to each Pod an IP inside the Azure subnet, so that Azure knows how to route the traffic.</li>
<li>Calico in VXLAN mode. Here Calico will wrap-up each paquet in another packet, the wrapper will only contain host IPs so that Azure knows how to route them. Then, when reaching the target Node, Calico unwraps the paquet to discover the real target IP, which would be a Pod IP located in the Calico subnet.</li>
</ol>
<p>In the below documentation, there are explanations on the tradeoff of each setup, in particular the Youtube video.</p>
<ul>
<li><a href="https://www.youtube.com/watch?v=JyLtg_SJ1lo&amp;list=PLoWxE_5hnZUZMWrEON3wxMBoIZvweGeiq&amp;index=8" rel="nofollow noreferrer">Youtube (9 min), Kubernetes networking on Azure</a></li>
<li>Calico-Azure: <a href="https://docs.projectcalico.org/reference/public-cloud/azure#about-calico-on-azure" rel="nofollow noreferrer">official site</a> and <a href="https://github.com/projectcalico/calico/blob/master/reference/public-cloud/azure.md" rel="nofollow noreferrer">Git</a></li>
<li><a href="https://docs.projectcalico.org/getting-started/kubernetes/installation/config-options" rel="nofollow noreferrer">Cutomizing Calico Maniest</a></li>
</ul>
<p><strong>Vocabulary:</strong></p>
<ul>
<li>CNI = Container network interface</li>
<li>IPAM = IP address management (to allocate IP addresses)</li>
</ul>
