<p>After installing just the basic Kubernetes packages and working with minikube, I have started just the basic kube-system pods. I'm trying to investigate why the kube-dns is not able to resolve domain names</p>

<p>Here are the versions I'm using</p>

<pre><code>Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:24:56 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:23:21 2018
  OS/Arch:          linux/amd64
  Experimental:     false

minikube version: v0.28.2
</code></pre>

<p>Kubectl:</p>

<pre><code>Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.2", GitCommit:"bb9ffb1654d4a729bb4cec18ff088eacc153c239", GitTreeState:"clean", BuildDate:"2018-08-07T23:17:28Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
</code></pre>

<p>Kubeadm:</p>

<pre><code>kubeadm version: &amp;version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
</code></pre>

<p>VirtualBox:</p>

<pre><code>Version 5.2.18 r124319 (Qt5.6.2)
</code></pre>

<p>Here are the system pods I have deployed:</p>

<pre><code>NAMESPACE     NAME                                    READY     STATUS    RESTARTS   AGE
default       busybox                                 1/1       Running   0          31m
kube-system   etcd-minikube                           1/1       Running   0          32m
kube-system   kube-addon-manager-minikube             1/1       Running   0          33m
kube-system   kube-apiserver-minikube                 1/1       Running   0          33m
kube-system   kube-controller-manager-minikube        1/1       Running   0          33m
kube-system   kube-dns-86f4d74b45-xjfmv               3/3       Running   2          33m
kube-system   kube-proxy-2kkzk                        1/1       Running   0          33m
kube-system   kube-scheduler-minikube                 1/1       Running   0          33m
kube-system   kubernetes-dashboard-5498ccf677-pz87g   1/1       Running   0          33m
kube-system   storage-provisioner                     1/1       Running   0          33m
</code></pre>

<p>I've also deployed busybox to allow me to execute commands inside  the containers</p>

<pre><code>kubectl exec busybox -- cat /etc/resolv.conf
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local mapleworks.com
options ndots:5
</code></pre>

<p>and </p>

<pre><code>kubectl exec busybox nslookup google.com
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

nslookup: can't resolve 'google.com'
command terminated with exit code 1
</code></pre>

<p>The same commands run on the VM itself yield the following:</p>

<pre><code>cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 127.0.1.1
search mapleworks.com  &lt;&lt;&lt; OUR local DNS server

nslookup google.com
Server:     127.0.1.1
Address:    127.0.1.1#53

Non-authoritative answer:
Name:   google.com
Address: 172.217.13.174
</code></pre>

<p>Questions:
kube-dns is using the default nameserver 10.96.0.10 whereas I would have expected the VM nameserver would have been imported into kubernetes.</p>

<p>While this same nameserver deployed on a native Windows or Mac platform is able to properly resolve domain names, this VM has an issue with it.</p>

<p>Is this some sort of Firewall issue as I've seen mentioned in some other posts?</p>

<p>I have inspected the kube-dns container logs but the most relevant are from the sidecar container.</p>

<pre><code>I0910 15:47:17.667100       1 main.go:51] Version v1.14.8
I0910 15:47:17.667195       1 server.go:45] Starting server (options {DnsMasqPort:53 DnsMasqAddr:127.0.0.1 DnsMasqPollIntervalMs:5000 Probes:[{Label:kubedns Server:127.0.0.1:10053 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:33} {Label:dnsmasq Server:127.0.0.1:53 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:33}] PrometheusAddr:0.0.0.0 PrometheusPort:10054 PrometheusPath:/metrics PrometheusNamespace:kubedns})
I0910 15:47:17.667240       1 dnsprobe.go:75] Starting dnsProbe {Label:kubedns Server:127.0.0.1:10053 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:33}
I0910 15:47:17.668244       1 dnsprobe.go:75] Starting dnsProbe {Label:dnsmasq Server:127.0.0.1:53 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:33}
W0910 15:50:04.780281       1 server.go:64] Error getting metrics from dnsmasq: read udp 127.0.0.1:34535-&gt;127.0.0.1:53: i/o timeout
W0910 15:50:11.781236       1 server.go:64] Error getting metrics from dnsmasq: read udp 127.0.0.1:50887-&gt;127.0.0.1:53: i/o timeout
W0910 15:50:24.844065       1 server.go:64] Error getting metrics from dnsmasq: read udp 127.0.0.1:52865-&gt;127.0.0.1:53: i/o timeout
W0910 15:50:31.845587       1 server.go:64] Error getting metrics from dnsmasq: read udp 127.0.0.1:42053-&gt;127.0.0.1:53: i/o timeout
</code></pre>

<p>The i/o timeouts correspond to the manual DNS query I've performed on google.com, I think</p>

<p>Otherwise I see here the localhost address and the port 53</p>

<p>I just don't know what is going on...</p>

<p>Each <code>kubelet</code> in a k8s cluster has <code>--cluster-dns</code> option. This option, in fact, provides a <code>Service</code> name for kube-dns <code>Deployment</code>. Each <code>kube-dns</code> Pod, in turn, has <code>dnsmasq</code> container, which is using a list of nameservers from the k8s node. You can check it in <code>dnsmasq</code> container's logs:</p>

<pre><code>I0720 03:49:51.081031       1 nanny.go:116] dnsmasq[13]: reading /etc/resolv.conf
I0720 03:49:51.081068       1 nanny.go:116] dnsmasq[13]: using nameserver 127.0.0.1#10053 for domain ip6.arpa 
I0720 03:49:51.081099       1 nanny.go:116] dnsmasq[13]: using nameserver 127.0.0.1#10053 for domain in-addr.arpa 
I0720 03:49:51.081130       1 nanny.go:116] dnsmasq[13]: using nameserver 127.0.0.1#10053 for domain cluster.local 
I0720 03:49:51.081160       1 nanny.go:116] dnsmasq[13]: using nameserver &lt;nameserver_1&gt;#53
I0720 03:49:51.081190       1 nanny.go:116] dnsmasq[13]: using nameserver &lt;nameserver_2&gt;#53
I0720 03:49:51.081222       1 nanny.go:116] dnsmasq[13]: using nameserver &lt;nameserver_N&gt;#53
</code></pre>

<p>When any <code>Pod</code> is created, by default, it has got <code>nameserver &lt;CLUSTER_DNS_IP&gt;</code> entry in <code>/etc/resolve.conf</code>. That's how any Pod can (or cannot) resolve certain domain name - through the <code>kube-dns</code> service.</p>

<p>For example, my cluster-dns is 10.233.0.3:</p>

<pre><code>$ kubectl -n test run -it --image=alpine:3.6 alpine -- sh                                                                      
If you don't see a command prompt, try pressing enter.
/ # cat /etc/resolv.conf 
nameserver 10.233.0.3
search test.svc.cluster.local svc.cluster.local cluster.local test.kz
/ # nslookup kubernetes-charts.storage.googleapis.com 10.233.0.3
Server:    10.233.0.3
Address 1: 10.233.0.3 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes-charts.storage.googleapis.com
Address 1: 74.125.131.128 lu-in-f128.1e100.net
Address 2: 2a00:1450:4010:c05::80 li-in-x80.1e100.net
</code></pre>

<p>So, if a <code>Node</code> (where the <code>kube-dns</code> is scheduled to) can resolve certain domain name, then any Pod can do the same.</p>
