<p>I'm trying to setup Kubernetes in Openstack + CoreOS. </p>

<p>I have master 10.240.63.84 and 2 minions .63 and .83. I also created 3 redis pods:</p>

<pre><code>redis-gopher-gziey        10.244.32.2     10.240.63.66/10.240.63.66
redis-managed-oh43e   10.244.32.3     10.240.63.66/10.240.63.66
redis-primary-fplln          10.244.54.2     10.240.63.83/10.240.63.83  
</code></pre>

<p>master's routing table looks like:</p>

<pre><code>10.240.63.0     *               255.255.255.0   U     0      0        0 eth0
10.240.63.1     *               255.255.255.255 UH    1024   0        0 eth0
10.244.0.0      *               255.255.0.0     U     0      0        0 flannel.1
10.244.50.0     *               255.255.255.0   U     0      0        0 docker0
</code></pre>

<p>and output of ifconfig -a is :</p>

<pre><code>docker0: flags=4099&lt;UP,BROADCAST,MULTICAST&gt;  mtu 1500
        inet 10.244.50.1  netmask 255.255.255.0  broadcast 0.0.0.0
        inet6 fe80::542f:6fff:fe4a:adf3  prefixlen 64  scopeid 0x20&lt;link&gt;
        ether 56:84:7a:fe:97:99  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1  bytes 90 (90.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0: flags=4163&lt;UP,BROADCAST,RUNNING,MULTICAST&gt;  mtu 1500  
        inet 10.240.63.84  netmask 255.255.255.0  broadcast 10.240.63.255
        inet6 fe80::f816:3eff:fe89:e9a0  prefixlen 64  scopeid 0x20&lt;link&gt;
        ether fa:16:3e:89:e9:a0  txqueuelen 1000  (Ethernet)
        RX packets 430706  bytes 559764129 (533.8 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 238519  bytes 116083693 (110.7 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

flannel.1: flags=4163&lt;UP,BROADCAST,RUNNING,MULTICAST&gt;  mtu 1450
        inet 10.244.50.0  netmask 255.255.0.0  broadcast 0.0.0.0
        inet6 fe80::601f:62ff:feed:1556  prefixlen 64  scopeid 0x20&lt;link&gt;
        ether 62:1f:62:ed:15:56  txqueuelen 0  (Ethernet)
        RX packets 20  bytes 1504 (1.4 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 79  bytes 7686 (7.5 KiB)
        TX errors 0  dropped 19 overruns 0  carrier 0  collisions 0
</code></pre>

<p>Flanneld config used for initialization is:</p>

<p>Master:</p>

<pre><code>    - name: flanneld.service
      command: start
      drop-ins:
        - name: 50-network-config.conf
          content: |
            [Service]
            ExecStartPre=/usr/bin/etcdctl set /coreos.com/network/config '{"Network":"10.244.0.0/16", "Backend": {"Type": "vxlan"}}'
            ExecStart=
            ExecStart=/usr/libexec/sdnotify-proxy /run/flannel/sd.sock \
                /usr/bin/docker run --net=host --privileged=true --rm \
                    --volume=/run/flannel:/run/flannel \
                    --env=NOTIFY_SOCKET=/run/flannel/sd.sock \
                    --env-file=/run/flannel/options.env \
                    --volume=${ETCD_SSL_DIR}:/etc/ssl/etcd:ro \
                    quay.io/coreos/flannel:${FLANNEL_VER} /opt/bin/flanneld --ip-masq=true --iface=eth0
</code></pre>

<p>Minion:</p>

<pre><code>    - name: flanneld.service
      command: start
      drop-ins:
        - name: 50-network-config.conf
          content: |
            [Service]
            ExecStartPre=/usr/bin/etcdctl set /coreos.com/network/config '{"Network":"10.244.0.0/16", "Backend": {"Type": "vxlan"}}'
            ExecStart=
            ExecStart=/usr/libexec/sdnotify-proxy /run/flannel/sd.sock \
                /usr/bin/docker run --net=host --privileged=true --rm \
                    --volume=/run/flannel:/run/flannel \
                    --env=NOTIFY_SOCKET=/run/flannel/sd.sock \
                    --env-file=/run/flannel/options.env \
                    --volume=${ETCD_SSL_DIR}:/etc/ssl/etcd:ro \
                    quay.io/coreos/flannel:${FLANNEL_VER} /opt/bin/flanneld -etcd-endpoints     http://10.240.63.84:4001 --ip-masq=true --iface=eth0
</code></pre>

<p>So the issue is that i can't ping any of the pods from master, as well as connect to any port, error is:</p>

<pre><code>ncat -v -t 10.244.32.2 6379
Ncat: Version 6.40 ( http://nmap.org/ncat )
Ncat: No route to host.
</code></pre>

<p>This sort of thing is hard to debug remotely.  Things I would check:</p>

<p>1) on the sender: <code>iptables -t raw -I OUTPUT -d 10.244.32.2 -j TRACE; dmesg -c &gt; /dev/null; ncat -v -t 10.244.32.2 6379; dmesg;</code></p>

<p>This will give you some insight into what the kernel is doing.</p>

<p>2) on the sender: <code>tcpdump -i any host 10.244.32.2</code> &amp; ncat -v -t 10.244.32.2 6379;`</p>

<p>This will give a bit more insight.</p>

<p>3) on the receiver: <code>iptables -t raw -I OUTPUT -d 10.244.32.2 -j TRACE; dmesg -c &gt; /dev/null; ncat -v -t 10.244.32.2 6379; dmesg;</code></p>

<p>This will tell you if the packet came through the encapsulation.</p>

<p>You need to basically prove the plumbing through the whole connection.</p>
