<p><strong>I have</strong></p>

<ul>
<li>Kubernetes: v.1.1.1</li>
<li>iptables v1.4.21 </li>
<li>kernel: 4.2.0-18-generic which come with Ubuntu wily</li>
<li>Networking is done via L2 VLAN terminated on switch</li>
<li>no cloud provider </li>
</ul>

<p><strong>what I do</strong></p>

<p>I'm experimenting with iptables mode for kube-proxy. I have enabled it with <code>--proxy_mode=iptables</code> argument. It seems some rule is missing:</p>

<pre><code>iptables -t nat -nvL

Chain PREROUTING (policy ACCEPT 8 packets, 459 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 2116  120K KUBE-SERVICES  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 2 packets, 120 bytes)
 pkts bytes target     prot opt in     out     source               destination         
  718 45203 KUBE-SERVICES  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */

Chain POSTROUTING (policy ACCEPT 5 packets, 339 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MASQUERADE  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes service traffic requiring SNAT */ mark match 0x4d415351

Chain KUBE-NODEPORTS (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/docker-registry-fe:tcp */ tcp dpt:31195 MARK set 0x4d415351
    0     0 KUBE-SVC-XZFGDLM7GMJHZHOY  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/docker-registry-fe:tcp */ tcp dpt:31195
    0     0 MARK       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* mngbox/jumpbox:ssh */ tcp dpt:30873 MARK set 0x4d415351
    0     0 KUBE-SVC-GLKZVFIDXOFHLJLC  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* mngbox/jumpbox:ssh */ tcp dpt:30873

Chain KUBE-SEP-5IXMK7UWPGVTWOJ7 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       10.116.160.8         0.0.0.0/0            /* mngbox/jumpbox:ssh */ MARK set 0x4d415351
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* mngbox/jumpbox:ssh */ tcp to:10.116.160.8:22

Chain KUBE-SEP-BNPLX5HQYOZINWEQ (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       10.116.161.6         0.0.0.0/0            /* kube-system/monitoring-influxdb:api */ MARK set 0x4d415351
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/monitoring-influxdb:api */ tcp to:10.116.161.6:8086

Chain KUBE-SEP-CJMHKLXPTJLTE3OP (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       10.116.254.2         0.0.0.0/0            /* default/kubernetes: */ MARK set 0x4d415351
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/kubernetes: */ tcp to:10.116.254.2:6443


Chain KUBE-SEP-GSM3BZTEXEBWDXPN (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       10.116.160.7         0.0.0.0/0            /* kube-system/kube-dns:dns */ MARK set 0x4d415351
    0     0 DNAT       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns */ udp to:10.116.160.7:53


Chain KUBE-SEP-OAYOAJINXRPUQDA3 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       10.116.160.7         0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */ MARK set 0x4d415351
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */ tcp to:10.116.160.7:53

Chain KUBE-SEP-PJJZDQNXDGWM7MU6 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       10.116.160.5         0.0.0.0/0            /* default/docker-registry-fe:tcp */ MARK set 0x4d415351
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/docker-registry-fe:tcp */ tcp to:10.116.160.5:443

Chain KUBE-SEP-RWODGLKOVWXGOHUR (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       10.116.161.6         0.0.0.0/0            /* kube-system/monitoring-influxdb:http */ MARK set 0x4d415351
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/monitoring-influxdb:http */ tcp to:10.116.161.6:8083

Chain KUBE-SEP-WE3Z7KMHA6KPJWKK (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       10.116.161.6         0.0.0.0/0            /* kube-system/monitoring-grafana: */ MARK set 0x4d415351
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/monitoring-grafana: */ tcp to:10.116.161.6:8080

Chain KUBE-SEP-YBQVM4LA4YMMZIWH (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       10.116.161.3         0.0.0.0/0            /* kube-system/monitoring-heapster: */ MARK set 0x4d415351
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/monitoring-heapster: */ tcp to:10.116.161.3:8082

Chain KUBE-SEP-YMZS7BLP4Y6MWTX5 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       10.116.160.9         0.0.0.0/0            /* infra/docker-registry-backend:docker-registry-backend */ MARK set 0x4d415351
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* infra/docker-registry-backend:docker-registry-backend */ tcp to:10.116.160.9:5000

Chain KUBE-SEP-ZDOOYAKDERKR43R3 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       10.116.160.10        0.0.0.0/0            /* default/kibana-logging: */ MARK set 0x4d415351
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/kibana-logging: */ tcp to:10.116.160.10:5601

Chain KUBE-SERVICES (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 KUBE-SVC-JRXTEHDDTAFMSEAS  tcp  --  *      *       0.0.0.0/0            10.116.0.48          /* kube-system/monitoring-grafana: cluster IP */ tcp dpt:80
    0     0 KUBE-SVC-CK6HVV5A27TDFNIA  tcp  --  *      *       0.0.0.0/0            10.116.0.188         /* kube-system/monitoring-influxdb:api cluster IP */ tcp dpt:8086
    0     0 KUBE-SVC-DKEW3YDJFV3YJLS2  tcp  --  *      *       0.0.0.0/0            10.116.0.6           /* infra/docker-registry-backend:docker-registry-backend cluster IP */ tcp dpt:5000
    0     0 KUBE-SVC-TCOU7JCQXEZGVUNU  udp  --  *      *       0.0.0.0/0            10.116.0.2           /* kube-system/kube-dns:dns cluster IP */ udp dpt:53
    0     0 KUBE-SVC-WEHLQ23XZWSA5ZX3  tcp  --  *      *       0.0.0.0/0            10.116.0.188         /* kube-system/monitoring-influxdb:http cluster IP */ tcp dpt:8083
    0     0 KUBE-SVC-XZFGDLM7GMJHZHOY  tcp  --  *      *       0.0.0.0/0            10.116.1.142         /* default/docker-registry-fe:tcp cluster IP */ tcp dpt:443
    0     0 MARK       tcp  --  *      *       0.0.0.0/0            10.116.254.3         /* default/docker-registry-fe:tcp external IP */ tcp dpt:443 MARK set 0x4d415351
    0     0 KUBE-SVC-XZFGDLM7GMJHZHOY  tcp  --  *      *       0.0.0.0/0            10.116.254.3         /* default/docker-registry-fe:tcp external IP */ tcp dpt:443 PHYSDEV match ! --physdev-is-in ADDRTYPE match src-type !LOCAL
    0     0 KUBE-SVC-XZFGDLM7GMJHZHOY  tcp  --  *      *       0.0.0.0/0            10.116.254.3         /* default/docker-registry-fe:tcp external IP */ tcp dpt:443 ADDRTYPE match dst-type LOCAL
    0     0 KUBE-SVC-ERIFXISQEP7F7OF4  tcp  --  *      *       0.0.0.0/0            10.116.0.2           /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:53
    0     0 KUBE-SVC-7IHGTXJ4CF2KVXJZ  tcp  --  *      *       0.0.0.0/0            10.116.1.126         /* kube-system/monitoring-heapster: cluster IP */ tcp dpt:80
    0     0 KUBE-SVC-GLKZVFIDXOFHLJLC  tcp  --  *      *       0.0.0.0/0            10.116.1.175         /* mngbox/jumpbox:ssh cluster IP */ tcp dpt:2345
    0     0 MARK       tcp  --  *      *       0.0.0.0/0            10.116.254.3         /* mngbox/jumpbox:ssh external IP */ tcp dpt:2345 MARK set 0x4d415351
    0     0 KUBE-SVC-GLKZVFIDXOFHLJLC  tcp  --  *      *       0.0.0.0/0            10.116.254.3         /* mngbox/jumpbox:ssh external IP */ tcp dpt:2345 PHYSDEV match ! --physdev-is-in ADDRTYPE match src-type !LOCAL
    0     0 KUBE-SVC-GLKZVFIDXOFHLJLC  tcp  --  *      *       0.0.0.0/0            10.116.254.3         /* mngbox/jumpbox:ssh external IP */ tcp dpt:2345 ADDRTYPE match dst-type LOCAL
    0     0 KUBE-SVC-6N4SJQIF3IX3FORG  tcp  --  *      *       0.0.0.0/0            10.116.0.1           /* default/kubernetes: cluster IP */ tcp dpt:443
    0     0 KUBE-SVC-B6ZEWWY2BII6JG2L  tcp  --  *      *       0.0.0.0/0            10.116.0.233         /* default/kibana-logging: cluster IP */ tcp dpt:8888
    0     0 MARK       tcp  --  *      *       0.0.0.0/0            10.116.254.3         /* default/kibana-logging: external IP */ tcp dpt:8888 MARK set 0x4d415351
    0     0 KUBE-SVC-B6ZEWWY2BII6JG2L  tcp  --  *      *       0.0.0.0/0            10.116.254.3         /* default/kibana-logging: external IP */ tcp dpt:8888 PHYSDEV match ! --physdev-is-in ADDRTYPE match src-type !LOCAL
    0     0 KUBE-SVC-B6ZEWWY2BII6JG2L  tcp  --  *      *       0.0.0.0/0            10.116.254.3         /* default/kibana-logging: external IP */ tcp dpt:8888 ADDRTYPE match dst-type LOCAL
    0     0 KUBE-NODEPORTS  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL

Chain KUBE-SVC-6N4SJQIF3IX3FORG (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 KUBE-SEP-CJMHKLXPTJLTE3OP  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/kubernetes: */

Chain KUBE-SVC-7IHGTXJ4CF2KVXJZ (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 KUBE-SEP-YBQVM4LA4YMMZIWH  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/monitoring-heapster: */

Chain KUBE-SVC-B6ZEWWY2BII6JG2L (3 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 KUBE-SEP-ZDOOYAKDERKR43R3  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/kibana-logging: */

Chain KUBE-SVC-CK6HVV5A27TDFNIA (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 KUBE-SEP-BNPLX5HQYOZINWEQ  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/monitoring-influxdb:api */

Chain KUBE-SVC-DKEW3YDJFV3YJLS2 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 KUBE-SEP-YMZS7BLP4Y6MWTX5  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* infra/docker-registry-backend:docker-registry-backend */

Chain KUBE-SVC-ERIFXISQEP7F7OF4 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 KUBE-SEP-OAYOAJINXRPUQDA3  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */

Chain KUBE-SVC-GLKZVFIDXOFHLJLC (4 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 KUBE-SEP-5IXMK7UWPGVTWOJ7  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* mngbox/jumpbox:ssh */

Chain KUBE-SVC-JRXTEHDDTAFMSEAS (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 KUBE-SEP-WE3Z7KMHA6KPJWKK  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/monitoring-grafana: */

Chain KUBE-SVC-TCOU7JCQXEZGVUNU (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 KUBE-SEP-GSM3BZTEXEBWDXPN  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns */

Chain KUBE-SVC-WEHLQ23XZWSA5ZX3 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 KUBE-SEP-RWODGLKOVWXGOHUR  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/monitoring-influxdb:http */

Chain KUBE-SVC-XZFGDLM7GMJHZHOY (4 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 KUBE-SEP-PJJZDQNXDGWM7MU6  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/docker-registry-fe:tcp */
</code></pre>

<p>When I do request to the service ip, in my case it's 10.116.0.2 I got an error</p>

<pre><code>;; connection timed out; no servers could be reached
</code></pre>

<p>while when I do request to the 10.116.160.7 server it's working fine. 
I can see that traffic is not directed to kube-proxy rules at all, so there is something missing probably. </p>

<p>I will highly appreciate any hint about missing rule</p>

<p><em>EDIT</em>
Ive updated my initial request with missing information requested by thokin, he pointed to the really good way to debug the iptables rules for kube-proxy, and I could identify my problem with:</p>

<pre><code>for c in PREROUTING OUTPUT POSTROUTING; do iptables -t nat -I $c -d 10.116.160.7 -j LOG --log-prefix "DBG@$c: "; done
for c in PREROUTING OUTPUT POSTROUTING; do iptables -t nat -I $c -d 10.116.0.2 -j LOG --log-prefix "DBG@$c: "; done
</code></pre>

<p>Then I've executed following commands:
    # nslookup kubernetes.default.svc.psc01.cluster  10.116.160.7
    Server:         10.116.160.7
    Address:        10.116.160.7#53</p>

<pre><code>Name:   kubernetes.default.svc.psc01.cluster
Address: 10.116.0.1

# nslookup kubernetes.default.svc.psc01.cluster  10.116.0.2
;; connection timed out; no servers could be reached
</code></pre>

<p>As a result I've got different "source" address and outgoing interface:</p>

<pre><code>[701768.263847] DBG@OUTPUT: IN= OUT=bond1.300 SRC=10.116.250.252 DST=10.116.0.2 LEN=82 TOS=0x00 PREC=0x00 TTL=64 ID=12436 PROTO=UDP SPT=54501 DPT=53 LEN=62 
[702620.454211] DBG@OUTPUT: IN= OUT=docker0 SRC=10.116.176.1 DST=10.116.160.7 LEN=82 TOS=0x00 PREC=0x00 TTL=64 ID=22733 PROTO=UDP SPT=28704 DPT=53 LEN=62 
[702620.454224] DBG@POSTROUTING: IN= OUT=docker0 SRC=10.116.176.1 DST=10.116.160.7 LEN=82 TOS=0x00 PREC=0x00 TTL=64 ID=22733 PROTO=UDP SPT=28704 DPT=53 LEN=62 
[702626.318258] DBG@OUTPUT: IN= OUT=bond1.300 SRC=10.116.250.252 DST=10.116.0.2 LEN=82 TOS=0x00 PREC=0x00 TTL=64 ID=30608 PROTO=UDP SPT=39443 DPT=53 LEN=62 
[702626.318263] DBG@OUTPUT: IN= OUT=bond1.300 SRC=10.116.250.252 DST=10.116.0.2 LEN=82 TOS=0x00 PREC=0x00 TTL=64 ID=30608 PROTO=UDP SPT=39443 DPT=53 LEN=62 
[702626.318266] DBG@OUTPUT: IN= OUT=bond1.300 SRC=10.116.250.252 DST=10.116.0.2 LEN=82 TOS=0x00 PREC=0x00 TTL=64 ID=30608 PROTO=UDP SPT=39443 DPT=53 LEN=62 
[702626.318270] DBG@OUTPUT: IN= OUT=bond1.300 SRC=10.116.250.252 DST=10.116.0.2 LEN=82 TOS=0x00 PREC=0x00 TTL=64 ID=30608 PROTO=UDP SPT=39443 DPT=53 LEN=62 
[702626.318284] DBG@POSTROUTING: IN= OUT=docker0 SRC=10.116.250.252 DST=10.116.160.7 LEN=82 TOS=0x00 PREC=0x00 TTL=64 ID=30608 PROTO=UDP SPT=39443 DPT=53 LEN=62 
</code></pre>

<p>So, by adding the route </p>

<pre><code>ip route add 10.116.0.0/23 dev docker0
</code></pre>

<p>Now it's working fine!</p>

<p>For future, the results of <code>iptables-save</code> are much easier to read (to me anyway).</p>

<p>I don't see anything missing here.</p>

<p><code>KUBE-SERVICES</code> traps 10.116.0.2 port 53/UDP and passes it to <code>KUBE-SVC-TCOU7JCQXEZGVUNU</code></p>

<p><code>KUBE-SVC-TCOU7JCQXEZGVUNU</code> has just one endpoint so jumps to <code>KUBE-SEP-GSM3BZTEXEBWDXPN</code></p>

<p><code>KUBE-SEP-GSM3BZTEXEBWDXPN</code> DNATs to 10.116.160.7 port 53/UDP</p>

<p>If you assert that 10.116.160.7 works while 10.116.0.2 does not, that is strange indeed.  It suggests that the iptables rules are not triggering at all.  Are you testing from the node itself or from a container?</p>

<p>What networking are you using?  L3 (underlay?) Flannel? OVS?  Something else?</p>

<p>What cloud provider (if any)?</p>

<p>First step to debug: run: <code>for c in PREROUTING OUTPUT; do iptables -t nat -I $c -d 10.116.0.2 -j LOG --log-prefix "DBG@$c: "; done</code></p>

<p>That will log any packets that iptables sees to your service IP.  Now look at <code>dmesg</code>.</p>
