<p>I setup k8s by <a href="https://github.com/kubernetes/kube-deploy/tree/master/docker-multinode" rel="nofollow">docker-multinode</a></p>

<pre><code>$ https_proxy=http://10.25.30.127:7777 IP_ADDRESS=10.25.24.116 MASTER_IP=10.25.30.127 ./worker.sh
+++ [0828 15:38:35] K8S_VERSION is set to: v1.3.6
+++ [0828 15:38:35] ETCD_VERSION is set to: 3.0.4
+++ [0828 15:38:35] FLANNEL_VERSION is set to: v0.6.1
+++ [0828 15:38:35] FLANNEL_IPMASQ is set to: true
+++ [0828 15:38:35] FLANNEL_NETWORK is set to: 10.1.0.0/16
+++ [0828 15:38:35] FLANNEL_BACKEND is set to: udp
+++ [0828 15:38:35] RESTART_POLICY is set to: unless-stopped
+++ [0828 15:38:35] MASTER_IP is set to: 10.25.30.127
+++ [0828 15:38:35] ARCH is set to: amd64
+++ [0828 15:38:35] IP_ADDRESS is set to: 10.25.24.116
+++ [0828 15:38:35] USE_CNI is set to: false
+++ [0828 15:38:35] --------------------------------------------
+++ [0828 15:38:35] Killing all kubernetes containers...
+++ [0828 15:38:35] Launching docker bootstrap...
+++ [0828 15:38:36] Launching flannel...
b70f4cc14aac8315740a916fae459d0b354280d97d8da743c67e9e692beea601
+++ [0828 15:38:37] FLANNEL_SUBNET is set to: 10.1.102.1/24
+++ [0828 15:38:37] FLANNEL_MTU is set to: 1472

$ k cluster-info
Kubernetes master is running at http://10.25.30.127:8080
KubeDNS is running at http://10.25.30.127:8080/api/v1/proxy/namespaces/kube-system/services/kube-dns
kubernetes-dashboard is running at http://10.25.30.127:8080/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard

$ k get nodes
NAME              STATUS    AGE
10.25.17.232      Ready     9h
10.25.19.197      Ready     3h
10.25.24.116      Ready     7h
139.1.1.1   Ready     9h
</code></pre>

<p>DNS started properly</p>

<pre><code>$ k logs kube-dns-v17.1-qaygj -c kubedns --namespace=kube-system |head -50
I0828 04:49:14.079474       1 server.go:91] Using https://10.0.0.1:443 for kubernetes master
I0828 04:49:14.081339       1 server.go:92] Using kubernetes API &lt;nil&gt;
I0828 04:49:14.081923       1 server.go:132] Starting SkyDNS server. Listening on port:10053
I0828 04:49:14.082071       1 server.go:139] skydns: metrics enabled on :/metrics
I0828 04:49:14.082181       1 dns.go:166] Waiting for service: default/kubernetes
I0828 04:49:14.082462       1 logs.go:41] skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053 [rcache 0]
I0828 04:49:14.082607       1 logs.go:41] skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053 [rcache 0]
I0828 04:49:14.480396       1 server.go:101] Setting up Healthz Handler(/readiness, /cache) on port :8081
I0828 04:49:14.483012       1 dns.go:660] DNS Record:&amp;{10.0.0.1 0 10 10  false 30 0  }, hash:24c3d825
I0828 04:49:14.483065       1 dns.go:660] DNS Record:&amp;{kubernetes.default.svc.cluster.local. 443 10 10  false 30 0  }, hash:c3f6ae26
I0828 04:49:14.483115       1 dns.go:660] DNS Record:&amp;{kubernetes.default.svc.cluster.local. 0 10 10  false 30 0  }, hash:b9b7d845
I0828 04:49:14.483160       1 dns.go:660] DNS Record:&amp;{10.0.0.24 0 10 10  false 30 0  }, hash:d8b58e70
I0828 04:49:14.483194       1 dns.go:660] DNS Record:&amp;{kubernetes-dashboard.kube-system.svc.cluster.local. 0 10 10  false 30 0  }, hash:529066a8
I0828 04:49:14.483237       1 dns.go:660] DNS Record:&amp;{10.0.0.10 0 10 10  false 30 0  }, hash:2d9aa69
I0828 04:49:14.483266       1 dns.go:660] DNS Record:&amp;{kube-dns.kube-system.svc.cluster.local. 53 10 10  false 30 0  }, hash:fdbb4e78
I0828 04:49:14.483309       1 dns.go:660] DNS Record:&amp;{kube-dns.kube-system.svc.cluster.local. 53 10 10  false 30 0  }, hash:fdbb4e78
I0828 04:49:14.483337       1 dns.go:660] DNS Record:&amp;{kube-dns.kube-system.svc.cluster.local. 0 10 10  false 30 0  }, hash:d1247c4e
I0828 04:49:16.678334       1 dns.go:439] Received DNS Request:kubernetes.default.svc.cluster.local., exact:false
I0828 04:49:16.678405       1 dns.go:539] records:[0xc820356af0], retval:[{10.0.0.1 0 10 10  false 30 0  /skydns/local/cluster/svc/default/kubernetes/3234633364383235}], path:[local cluster svc default kubernetes]
I0828 04:49:16.777991       1 dns.go:439] Received DNS Request:kubernetes.default.svc.cluster.local., exact:false
I0828 04:49:16.778100       1 dns.go:539] records:[0xc820356af0], retval:[{10.0.0.1 0 10 10  false 30 0  /skydns/local/cluster/svc/default/kubernetes/3234633364383235}], path:[local cluster svc default kubernetes]
I0828 04:49:16.778886       1 dns.go:583] Received ReverseRecord Request:1.0.0.10.in-addr.arpa.
I0828 04:49:46.778352       1 dns.go:439] Received DNS Request:kubernetes.default.svc.cluster.local., exact:false
I0828 04:49:46.778406       1 dns.go:539] records:[0xc820356af0], retval:[{10.0.0.1 0 10 10  false 30 0  /skydns/local/cluster/svc/default/kubernetes/3234633364383235}], path:[local cluster svc default kubernetes]
I0828 04:49:46.778932       1 dns.go:583] Received ReverseRecord Request:1.0.0.10.in-addr.arpa.
I0828 04:50:16.879611       1 dns.go:439] Received DNS Request:kubernetes.default.svc.cluster.local., exact:false
I0828 04:50:16.879685       1 dns.go:539] records:[0xc820356af0], retval:[{10.0.0.1 0 10 10  false 30 0  /skydns/local/cluster/svc/default/kubernetes/3234633364383235}], path:[local cluster svc default kubernetes]
I0828 04:50:16.880274       1 dns.go:439] Received DNS Request:kubernetes.default.svc.cluster.local., exact:false
I0828 04:50:16.880332       1 dns.go:539] records:[0xc820356af0], retval:[{10.0.0.1 0 10 10  false 30 0  /skydns/local/cluster/svc/default/kubernetes/3234633364383235}], path:[local cluster svc default kubernetes]
I0828 04:50:16.880900       1 dns.go:583] Received ReverseRecord Request:1.0.0.10.in-addr.arpa.
I0828 04:50:46.878037       1 dns.go:439] Received DNS Request:kubernetes.default.svc.cluster.local., exact:false
I0828 04:50:46.878094       1 dns.go:539] records:[0xc820356af0], retval:[{10.0.0.1 0 10 10  false 30 0  /skydns/local/cluster/svc/default/kubernetes/3234633364383235}], path:[local cluster svc default kubernetes]
I0828 04:50:46.978007       1 dns.go:583] Received ReverseRecord Request:1.0.0.10.in-addr.arpa.
I0828 04:51:16.778397       1 dns.go:439] Received DNS Request:kubernetes.default.svc.cluster.local., exact:false
I0828 04:51:16.778455       1 dns.go:539] records:[0xc820356af0], retval:[{10.0.0.1 0 10 10  false 30 0  /skydns/local/cluster/svc/default/kubernetes/3234633364383235}], path:[local cluster svc default kubernetes]
I0828 04:51:16.779062       1 dns.go:439] Received DNS Request:kubernetes.default.svc.cluster.local., exact:false
I0828 04:51:16.779110       1 dns.go:539] records:[0xc820356af0], retval:[{10.0.0.1 0 10 10  false 30 0  /skydns/local/cluster/svc/default/kubernetes/3234633364383235}], path:[local cluster svc default kubernetes]
I0828 04:51:16.779588       1 dns.go:583] Received ReverseRecord Request:1.0.0.10.in-addr.arpa.
I0828 04:51:46.778319       1 dns.go:439] Received DNS Request:kubernetes.default.svc.cluster.local., exact:false
I0828 04:51:46.778374       1 dns.go:539] records:[0xc820356af0], retval:[{10.0.0.1 0 10 10  false 30 0  /skydns/local/cluster/svc/default/kubernetes/3234633364383235}], path:[local cluster svc default kubernetes]
I0828 04:51:46.779048       1 dns.go:583] Received ReverseRecord Request:1.0.0.10.in-addr.arpa.
I0828 04:52:16.878240       1 dns.go:439] Received DNS Request:kubernetes.default.svc.cluster.local., exact:false
I0828 04:52:16.878309       1 dns.go:539] records:[0xc820356af0], retval:[{10.0.0.1 0 10 10  false 30 0  /skydns/local/cluster/svc/default/kubernetes/3234633364383235}], path:[local cluster svc default kubernetes]
I0828 04:52:16.878848       1 dns.go:439] Received DNS Request:kubernetes.default.svc.cluster.local., exact:false
I0828 04:52:16.878886       1 dns.go:539] records:[0xc820356af0], retval:[{10.0.0.1 0 10 10  false 30 0  /skydns/local/cluster/svc/default/kubernetes/3234633364383235}], path:[local cluster svc default kubernetes]
I0828 04:52:16.977642       1 dns.go:583] Received ReverseRecord Request:1.0.0.10.in-addr.arpa.
I0828 04:52:46.678628       1 dns.go:439] Received DNS Request:kubernetes.default.svc.cluster.local., exact:false
I0828 04:52:46.678685       1 dns.go:539] records:[0xc820356af0], retval:[{10.0.0.1 0 10 10  false 30 0  /skydns/local/cluster/svc/default/kubernetes/3234633364383235}], path:[local cluster svc default kubernetes]
I0828 04:52:46.878096       1 dns.go:583] Received ReverseRecord Request:1.0.0.10.in-addr.arpa.
I0828 04:53:16.679056       1 dns.go:439] Received DNS Request:kubernetes.default.svc.cluster.local., exact:false
</code></pre>

<p>When I start guestbook I notice the dns log</p>

<pre><code>I0828 14:39:07.977746       1 dns.go:660] DNS Record:&amp;{redis-master.default.svc.cluster.local. 0 10 10  false 30 0  }, hash:606d0bd5
I0828 14:39:14.178122       1 dns.go:660] DNS Record:&amp;{10.0.0.147 0 10 10  false 30 0  }, hash:e8b710be
I0828 14:39:14.178212       1 dns.go:660] DNS Record:&amp;{myapp.default.svc.cluster.local. 0 10 10  false 30 0  }, hash:2077e618
I0828 14:39:14.178254       1 dns.go:660] DNS Record:&amp;{10.0.0.145 0 10 10  false 30 0  }, hash:d93e9f2c
I0828 14:39:14.178296       1 dns.go:660] DNS Record:&amp;{redis-master.default.svc.cluster.local. 0 10 10  false 30 0  }, hash:606d0bd5
I0828 14:39:14.178336       1 dns.go:660] DNS Record:&amp;{10.0.0.171 0 10 10  false 30 0  }, hash:cb7a4915
I0828 14:39:14.178364       1 dns.go:660] DNS Record:&amp;{redis-slave.default.svc.cluster.local. 0 10 10  false 30 0  }, hash:ff7d45d8
I0828 14:39:14.178404       1 dns.go:660] DNS Record:&amp;{10.0.0.139 0 10 10  false 30 0  }, hash:9afc24c9
I0828 14:39:14.178431       1 dns.go:660] DNS Record:&amp;{guestbook.default.svc.cluster.local. 0 10 10  false 30 0  }, hash:78248924
I0828 14:39:14.178456       1 dns.go:660] DNS Record:&amp;{10.0.0.1 0 10 10  false 30 0  }, hash:24c3d825
I0828 14:39:14.178513       1 dns.go:660] DNS Record:&amp;{kubernetes.default.svc.cluster.local. 443 10 10  false 30 0  }, hash:c3f6ae26
</code></pre>

<p>But dns lookup is not working</p>

<p>Guestbook info report</p>

<pre><code>PANIC: dial tcp: lookup redis-master on 10.0.0.10:53: read udp 10.1.102.2:46755-&gt;10.0.0.10:53: i/o timeout
goroutine 277 [running]:
github.com/codegangsta/negroni.(*Recovery).ServeHTTP.func1(0x7f1d98e5ba90, 0xc820323580, 0xc8200b8ec0)
    /go/src/github.com/codegangsta/negroni/recovery.go:34 +0xe9
panic(0x7a8c60, 0xc820327ae0)
</code></pre>

<p>Guestbook env page</p>

<pre><code>{
  "GUESTBOOK_PORT": "tcp://10.0.0.139:3000",
  "GUESTBOOK_PORT_3000_TCP": "tcp://10.0.0.139:3000",
  "GUESTBOOK_PORT_3000_TCP_ADDR": "10.0.0.139",
  "GUESTBOOK_PORT_3000_TCP_PORT": "3000",
  "GUESTBOOK_PORT_3000_TCP_PROTO": "tcp",
  "GUESTBOOK_SERVICE_HOST": "10.0.0.139",
  "GUESTBOOK_SERVICE_PORT": "3000",
  "HOME": "/",
  "HOSTNAME": "guestbook-advba",
  "KUBERNETES_PORT": "tcp://10.0.0.1:443",
  "KUBERNETES_PORT_443_TCP": "tcp://10.0.0.1:443",
  "KUBERNETES_PORT_443_TCP_ADDR": "10.0.0.1",
  "KUBERNETES_PORT_443_TCP_PORT": "443",
  "KUBERNETES_PORT_443_TCP_PROTO": "tcp",
  "KUBERNETES_SERVICE_HOST": "10.0.0.1",
  "KUBERNETES_SERVICE_PORT": "443",
  "KUBERNETES_SERVICE_PORT_HTTPS": "443",
  "MYAPP_PORT": "tcp://10.0.0.147:8765",
  "MYAPP_PORT_8765_TCP": "tcp://10.0.0.147:8765",
  "MYAPP_PORT_8765_TCP_ADDR": "10.0.0.147",
  "MYAPP_PORT_8765_TCP_PORT": "8765",
  "MYAPP_PORT_8765_TCP_PROTO": "tcp",
  "MYAPP_SERVICE_HOST": "10.0.0.147",
  "MYAPP_SERVICE_PORT": "8765",
  "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
  "REDIS_MASTER_PORT": "tcp://10.0.0.35:6379",
  "REDIS_MASTER_PORT_6379_TCP": "tcp://10.0.0.35:6379",
  "REDIS_MASTER_PORT_6379_TCP_ADDR": "10.0.0.35",
  "REDIS_MASTER_PORT_6379_TCP_PORT": "6379",
  "REDIS_MASTER_PORT_6379_TCP_PROTO": "tcp",
  "REDIS_MASTER_SERVICE_HOST": "10.0.0.35",
  "REDIS_MASTER_SERVICE_PORT": "6379",
  "REDIS_SLAVE_PORT": "tcp://10.0.0.171:6379",
  "REDIS_SLAVE_PORT_6379_TCP": "tcp://10.0.0.171:6379",
  "REDIS_SLAVE_PORT_6379_TCP_ADDR": "10.0.0.171",
  "REDIS_SLAVE_PORT_6379_TCP_PORT": "6379",
  "REDIS_SLAVE_PORT_6379_TCP_PROTO": "tcp",
  "REDIS_SLAVE_SERVICE_HOST": "10.0.0.171",
  "REDIS_SLAVE_SERVICE_PORT": "6379"
}
</code></pre>

<p>I also try to resolve in dns pod</p>

<pre><code>$ k get svc --namespace=kube-system
NAME                   CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE
kube-dns               10.0.0.10    &lt;none&gt;        53/UDP,53/TCP   10h
kubernetes-dashboard   10.0.0.24    &lt;none&gt;        80/TCP          10h
$ k exec -it kube-dns-v17.1-qaygj -c kubedns --namespace=kube-system -- /bin/sh
/ # nslookup redis-master
Server:    10.143.22.118
Address 1: 10.143.22.118

nslookup: can't resolve 'redis-master'
/ # nslookup baidu.com
Server:    10.143.22.118
Address 1: 10.143.22.118

Name:      baidu.com
Address 1: 111.13.101.208
Address 2: 220.181.57.217
Address 3: 123.125.114.144
Address 4: 180.149.132.47
/ # nslookup redis-master 10.0.0.10
Server:    10.0.0.10
Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local

nslookup: can't resolve 'redis-master'
/ # nslookup baidu.com 10.0.0.10
Server:    10.0.0.10
Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local

Name:      baidu.com
Address 1: 123.125.114.144
Address 2: 180.149.132.47
Address 3: 111.13.101.208
Address 4: 220.181.57.217
</code></pre>

<p>Why kube-dns not works ?</p>

<p>This problem is caused by the ip conflict, provider use 10 as internal network, docker-multinode use 10/8 as default cluster ip range, then something got wired.</p>

<p>This is what I done</p>

<ul>
<li>Copy /etc/kubernetes outside the container</li>
<li>Replace <code>10.0.0.1</code> to <code>172.16.0.1</code> in <code>/etc/kubernetes/master-multi/master-multi.json</code></li>
<li><p>Change <code>addon</code> volume in <code>/etc/kubernetes/master-multi/addon-manager.json</code> to </p>

<p>{
        "name": "addons",
        "hostPath": {"path":"/path/to/you/own/etc/kubernetes/"}
}</p></li>
<li><p>Change <code>clusterIP</code> in <code>/etc/kubernetes/addon/skydns-svc.yaml</code> to <code>172.16.0.10</code></p></li>
<li>Change all <code>--cluster-dns=10.0.0.10</code> in <code>common.sh</code> to <code>--cluster-dns=172.16.0.10</code></li>
<li>Add <code>-v /path/to/you/own/etc/kubernetes/:/etc/kubernetes/ \</code> to KUBECTL_MOUNT in <code>common.sh</code></li>
<li>Then <code>FLANNEL_IPMASQ=false FLANNEL_NETWORK=172.16.0.0/16 ./master.sh</code></li>
<li>Done</li>
</ul>

<p>I also add an issues to kube-deploy <a href="https://github.com/kubernetes/kube-deploy/issues/215" rel="nofollow">kube-deploy#215</a></p>
