<p>Just getting started with Kubernetes.  I cannot seem to connect pods running on different nodes to communicate with each other.  </p>

<p>I set up a Kubernetes Cluster with Calico networking on three AWS EC2 instances (one master, two workers <a href="https://docs.projectcalico.org/reference/public-cloud/aws#routing-traffic-within-a-single-vpc-subnet" rel="nofollow noreferrer">all with src/dest check disabled as described by the Calico website</a>).  Each instance is using the same Security Group with all TCP/UDP/ICMP ports open for 10.0.0.0/8 and 192.168.0.0/16 to make sure there is no blocked ports inside my cluster. </p>

<p>using a vanilla repo install </p>

<pre><code>~$ sudo apt-get install -y docker.io kubelet kubeadm kubectl
~$ sudo kubeadm init --pod-network-cidr=192.168.0.0/16  
</code></pre>

<p>and basic Calico install</p>

<pre><code>~$ kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml
</code></pre>

<p>joined two worker nodes to the cluster</p>

<pre><code> sudo kubeadm join &lt;Master IP&gt;:6443 --token &lt;Token&gt;  --discovery-token-ca-cert-hash sha256:&lt;cert hash&gt; 
</code></pre>

<p>Once up and running, I created three replicas for testing:</p>

<pre><code>~$ kubectl run pingtest --image=busybox --replicas=3 -- sleep infinity
</code></pre>

<p>two on the first node and one on the second node</p>

<pre><code>~$ kubectl get pod -l run=pingtest -o wide
NAME                        READY   STATUS    RESTARTS   AGE   IP               NODE              NOMINATED NODE   READINESS GATES
pingtest-7689dd958f-9mfgl   1/1     Running   0          15m   192.168.218.65   ip-10-78-31-198   &lt;none&gt;           &lt;none&gt;
pingtest-7689dd958f-l288v   1/1     Running   0          15m   192.168.218.66   ip-10-78-31-198   &lt;none&gt;           &lt;none&gt;
pingtest-7689dd958f-z2l97   1/1     Running   0          15m   192.168.237.65   ip-10-78-11-83    &lt;none&gt;           &lt;none&gt;
</code></pre>

<p>log into a shell on the first pod</p>

<pre><code>~$ kubectl exec -ti pingtest-7689dd958f-9mfgl /bin/sh
</code></pre>

<p>When I ping pods on the same node everything works</p>

<pre><code>/ # ping 192.168.218.66 -c 2
PING 192.168.218.66 (192.168.218.66): 56 data bytes
64 bytes from 192.168.218.66: seq=0 ttl=63 time=0.105 ms
64 bytes from 192.168.218.66: seq=1 ttl=63 time=0.078 ms
</code></pre>

<p>but when I ping a pod on another node, no response </p>

<pre><code>/ # ping 192.168.237.65 -c 2
PING 192.168.237.65 (192.168.237.65): 56 data bytes

--- 192.168.237.65 ping statistics ---
2 packets transmitted, 0 packets received, 100% packet loss
</code></pre>

<p>What am I missing?  What is preventing communication between the pods on different nodes?</p>

<p>I figured out the issue. It was with the AWS configuration and some extra work you have to do in that environment. </p>

<ol>
<li>For the three AWS EC2 instances, they must all have src/dest check <strong>disabled</strong> <a href="https://i.stack.imgur.com/NMPXN.png" rel="nofollow noreferrer">as described by the Calico website</a>).</li>
<li>For the Security Group covering your AWS instances, you must add a <strong>Custom Protocol</strong> (not Custom TCP or Custom UDP), select 4 (IP in IP) in the protocol column and choose the subnets covering your instance (.eg. 10.0.0.0/8, 192.168.0.0/16). Then you can use the <code>curl</code> command to address you Pods/ServiceIP
<a href="https://i.stack.imgur.com/NMPXN.png" rel="nofollow noreferrer">AWS Security Group Settings</a></li>
</ol>
