<p>I setup a 3 nodes kubernetes (<code>v1.9.3</code>) cluster on Ubuntu 16.04. </p>

<p>Prior setup I cleared the iptables rules and follow k8s documents for flannel with following command to initialize the cluster:</p>

<pre><code># kubeadm init --apiserver-advertise-address 192.168.56.20 --pod-network-cidr=10.244.0.0/16 --kubernetes-version 1.9.3
# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml
</code></pre>

<p>The previous command seemed successful:</p>

<pre><code># kubectl -n kube-system -n kube-system get pods
NAME                             READY     STATUS    RESTARTS   AGE
etcd-master                      1/1       Running   0          3m
kube-apiserver-master            1/1       Running   0          2m
kube-controller-manager-master   1/1       Running   0          2m
kube-dns-6f4fd4bdf-4c76v         3/3       Running   0          3m
kube-flannel-ds-wbx97            1/1       Running   0          1m
kube-proxy-x65lv                 1/1       Running   0          3m
kube-scheduler-master            1/1       Running   0          2m 
</code></pre>

<p>But the problem is <code>kube-dns</code> seems got wrong service endpoint address assigned, this can be seen with following commands:</p>

<pre><code># kubectl get ep kube-dns --namespace=kube-system            
NAME       ENDPOINTS                     AGE
kube-dns   172.17.0.2:53,172.17.0.2:53   3m
root@master:~# kubectl describe service kube-dns -n kube-system           
Name:              kube-dns
Namespace:         kube-system
Labels:            k8s-app=kube-dns
                   kubernetes.io/cluster-service=true
                   kubernetes.io/name=KubeDNS
Annotations:       &lt;none&gt;
Selector:          k8s-app=kube-dns
Type:              ClusterIP
IP:                10.96.0.10
Port:              dns  53/UDP
TargetPort:        53/UDP
Endpoints:         172.17.0.2:53
Port:              dns-tcp  53/TCP
TargetPort:        53/TCP
Endpoints:         172.17.0.2:53
Session Affinity:  None
Events:            &lt;none&gt;
</code></pre>

<p>The <code>172.17.0.2</code> is the IP address assigned by docker bridge (<code>docker0</code>) for <code>kube-dns</code> container. On working k8s network setup, the <code>kube-dns</code> should have endpoints with address from <code>podSubnet</code> (<code>10.244.0.0/16</code>). </p>

<p>The effect of current setup is all the pods will not have functioned DNS while IP communication is ok. </p>

<p>I tried to delete <code>kube-dns</code> pod to see the new <code>kube-dns</code> containers can pick up the endpoints from <code>podSubnet</code> but they don't. </p>

<p>From the startup logs of 3 <code>kube-dns</code> containers, there are no ANY error messages. </p>

robert
<p>I think I have found out the root cause for this. It is the previous <code>kubeadm reset</code> did not remove both <code>cni</code> and <code>flannel.1</code> interfaces. So the next <code>kubeadm init</code> makes <code>kube-dns</code> believes the Kubernetes network plugin is already in place before I apply the flannel yaml.</p>

<p>After I check and remove any virtual NICs created by flannel plugin when tearing down kubernetes cluster, the next <code>kubeadm init</code> can succeed without this issue any more. </p>

<p>The same thing applies to <em>Weave Net</em> that requires to run <code>weave reset</code> to remove remained virtual weave NICs. </p>
