<p>I have a kubernetes cluster (using flannel):</p>

<pre><code>kubectl version
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"17c77c7898218073f14c8d573582e8d2313dc740", GitTreeState:"clean", BuildDate:"2018-10-24T06:54:59Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.5", GitCommit:"51dd616cdd25d6ee22c83a858773b607328a18ec", GitTreeState:"clean", BuildDate:"2019-01-16T18:14:49Z", GoVersion:"go1.10.7", Compiler:"gc", Platform:"linux/amd64"}

kubectl get cs
NAME                 STATUS    MESSAGE              ERROR
scheduler            Healthy   ok
controller-manager   Healthy   ok
etcd-0               Healthy   {"health": "true"}
</code></pre>

<p>Everything seems to be running okay</p>

<pre><code>$ kubectl get pods -n kube-system
NAME                            READY   STATUS    RESTARTS   AGE
coredns-576cbf47c7-q7ncm        1/1     Running   1          30m
coredns-576cbf47c7-tclp8        1/1     Running   1          30m
etcd-kube1                      1/1     Running   1          30m
kube-apiserver-kube1            1/1     Running   1          30m
kube-controller-manager-kube1   1/1     Running   1          30m
kube-flannel-ds-amd64-6vlkx     1/1     Running   1          30m
kube-flannel-ds-amd64-7twk8     1/1     Running   1          30m
kube-flannel-ds-amd64-rqzr7     1/1     Running   1          30m
kube-proxy-krfzk                1/1     Running   1          30m
kube-proxy-vrssw                1/1     Running   1          30m
kube-proxy-xlrgz                1/1     Running   1          30m
kube-scheduler-kube1            1/1     Running   1          30m
</code></pre>

<p>Now I've deployed 2 pods (without a service). 2 NGinx pods. I've also created a busybox pod. When I curl from inside the busybox pod to the nginx pod on the same node it works:</p>

<pre><code>kubectl get pods -o wide
NAME                   READY   STATUS    RESTARTS   AGE   IP           NODE    NOMINATED NODE
busybox                1/1     Running   2          30m   10.244.2.5   kube2   &lt;none&gt;
nginx-d55b94fd-l7swz   1/1     Running   1          30m   10.244.2.4   kube2   &lt;none&gt;
nginx-d55b94fd-zg7sj   1/1     Running   1          30m   10.244.1.6   kube3   &lt;none&gt;
</code></pre>

<p>curl:</p>

<pre><code>kubectl exec busybox -- curl 10.244.2.4
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   612  100   612    0     0   357k      0 --:--:-- --:--:-- --:--:--  597k
&lt;!DOCTYPE html&gt;
&lt;html&gt;
&lt;head&gt;
&lt;title&gt;Welcome to nginx!&lt;/title&gt;
&lt;style&gt;
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
&lt;/style&gt;
&lt;/head&gt;
&lt;body&gt;
&lt;h1&gt;Welcome to nginx!&lt;/h1&gt;
&lt;p&gt;If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.&lt;/p&gt;

&lt;p&gt;For online documentation and support please refer to
&lt;a href="http://nginx.org/"&gt;nginx.org&lt;/a&gt;.&lt;br/&gt;
Commercial support is available at
&lt;a href="http://nginx.com/"&gt;nginx.com&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Thank you for using nginx.&lt;/em&gt;&lt;/p&gt;
&lt;/body&gt;
&lt;/html&gt;
</code></pre>

<p>But when I curl the pod on a different node:</p>

<pre><code> kubectl exec busybox -- curl 10.244.1.6
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- 10.244.1.6 port 80: No route to host
</code></pre>

<p>How can I debug this? What can be wrong? (All firewalls are turned off/disabled)</p>

<p>additional info:</p>

<pre><code>$ sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1


$ sysctl net.bridge.bridge-nf-call-iptables
net.bridge.bridge-nf-call-iptables = 1
</code></pre>

<p>info:</p>

<pre><code>kubectl exec -it busybox ip a
1: lo: &lt;LOOPBACK,UP,LOWER_UP&gt; mtu 65536 qdisc noqueue qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
3: eth0@if7: &lt;BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN&gt; mtu 1450 qdisc noqueue
    link/ether 0a:58:0a:f4:02:05 brd ff:ff:ff:ff:ff:ff
    inet 10.244.2.5/24 scope global eth0
       valid_lft forever preferred_lft forever
</code></pre>

<p>Iptables:</p>

<pre><code>vagrant@kube1:~$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
KUBE-EXTERNAL-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes externally-visible service portals */
KUBE-FIREWALL  all  --  anywhere             anywhere

Chain FORWARD (policy DROP)
target     prot opt source               destination
KUBE-FORWARD  all  --  anywhere             anywhere             /* kubernetes forwarding rules */
DOCKER-USER  all  --  anywhere             anywhere
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  kube1/16             anywhere
ACCEPT     all  --  anywhere             kube1/16

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
KUBE-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes service portals */
KUBE-FIREWALL  all  --  anywhere             anywhere

Chain DOCKER (1 references)
target     prot opt source               destination

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target     prot opt source               destination
DROP       all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere

Chain DOCKER-USER (1 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere

Chain KUBE-EXTERNAL-SERVICES (1 references)
target     prot opt source               destination

Chain KUBE-FIREWALL (2 references)
target     prot opt source               destination
DROP       all  --  anywhere             anywhere             /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000

Chain KUBE-FORWARD (1 references)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere             /* kubernetes forwarding rules */ mark match 0x4000/0x4000
ACCEPT     all  --  kube1/16             anywhere             /* kubernetes forwarding conntrack pod source rule */ ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             kube1/16             /* kubernetes forwarding conntrack pod destination rule */ ctstate RELATED,ESTABLISHED

Chain KUBE-SERVICES (1 references)
target     prot opt source               destination
</code></pre>

<blockquote>
  <p>How can I debug this? What can be wrong? (All firewalls are turned off/disabled)</p>
</blockquote>

<p>This may be the problem if you have disabled iptables on your nodes. The overlay (flannel) sets up iptables to allow pod to pod traffic. You can check on your K8s nodes with something like this:</p>

<pre><code>iptables-save  | grep 10.244.2.4
iptables-save  | grep 10.244.2.5
iptables-save  | grep 10.244.2.6
</code></pre>

<p>You should see some rules like this for port <code>80</code>:</p>

<pre><code>-A KUBE-SEP-XXXXXXXXXXXXXXXX -s 10.244.2.4/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-XXXXXXXXXXXXXXXX -p tcp -m tcp -j DNAT --to-destination 10.244.2.4:80
</code></pre>
