<h1>What happened</h1>
<p>Resolving an external domain from within a pod fails with <strong>SERVFAIL</strong> message. In the logs, <strong>i/o timeout</strong> error is mentioned.</p>
<h1>What I expected to happen</h1>
<p>External domains should be successfully resolved from the pods.</p>
<h1>How to reproduce it</h1>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: Pod
metadata:
  name: dnsutils
  namespace: default
spec:
  containers:
  - name: dnsutils
    image: gcr.io/kubernetes-e2e-test-images/dnsutils:1.3
    command:
      - sleep
      - &quot;3600&quot;
    imagePullPolicy: IfNotPresent
  restartPolicy: Always
</code></pre>
<ol>
<li><p>Create the pod above (from <a href="https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/" rel="noreferrer">Debugging DNS Resolution</a> help page).</p>
</li>
<li><p>Run <code>kubectl exec dnsutils -it -- nslookup google.com</code></p>
<pre class="lang-sh prettyprint-override"><code>pig@pig202:~$ kubectl exec dnsutils -it -- nslookup google.com
Server:      10.152.183.10
Address: 10.152.183.10#53

** server can't find google.com.mshome.net: SERVFAIL

command terminated with exit code 1
</code></pre>
</li>
<li><p>Also run <code>kubectl exec dnsutils -it -- nslookup google.com.</code></p>
<pre class="lang-sh prettyprint-override"><code>pig@pig202:~$ kubectl exec dnsutils -it -- nslookup google.com.
Server:      10.152.183.10
Address: 10.152.183.10#53

** server can't find google.com: SERVFAIL

command terminated with exit code 1
</code></pre>
</li>
</ol>
<h1>Additional information</h1>
<p>I am using <strong>microk8s</strong> environment in a <strong>Hyper-V virtual machine</strong>.</p>
<p>Resolving DNS from the virtual machine works, and Kubernetes is able to pull container images. It's only from within the pods that the resolution is failing meaning I cannot communicate with the Internet from within the pods.</p>
<p>This is OK:</p>
<pre class="lang-sh prettyprint-override"><code>pig@pig202:~$ kubectl exec dnsutils -it -- nslookup kubernetes.default
Server:     10.152.183.10
Address:    10.152.183.10#53

Name:   kubernetes.default.svc.cluster.local
Address: 10.152.183.1
</code></pre>
<h1>Environment</h1>
<h3>The version of CoreDNS</h3>
<pre class="lang-yaml prettyprint-override"><code>image: 'coredns/coredns:1.6.6'
</code></pre>
<h3>Corefile (taken from ConfigMap)</h3>
<pre class="lang-yaml prettyprint-override"><code>  Corefile: |
    .:53 {
        errors
        health {
          lameduck 5s
        }
        ready
        log . {
          class error
        }
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . 8.8.8.8 8.8.4.4
        cache 30
        loop
        reload
        loadbalance
    }
</code></pre>
<h3>Logs</h3>
<pre class="lang-yaml prettyprint-override"><code>pig@pig202:~$ kubectl logs --namespace=kube-system -l k8s-app=kube-dns -f
[INFO] 10.1.99.26:47204 - 29832 &quot;AAAA IN grafana.com. udp 29 false 512&quot; NOERROR - 0 2.0002558s
[ERROR] plugin/errors: 2 grafana.com. AAAA: read udp 10.1.99.19:52008-&gt;8.8.8.8:53: i/o timeout
[INFO] 10.1.99.26:59350 - 50446 &quot;A IN grafana.com. udp 29 false 512&quot; NOERROR - 0 2.0002028s
[ERROR] plugin/errors: 2 grafana.com. A: read udp 10.1.99.19:60405-&gt;8.8.8.8:53: i/o timeout
[INFO] 10.1.99.26:43050 - 13676 &quot;AAAA IN grafana.com. udp 29 false 512&quot; NOERROR - 0 2.0002151s
[ERROR] plugin/errors: 2 grafana.com. AAAA: read udp 10.1.99.19:45624-&gt;8.8.8.8:53: i/o timeout
[INFO] 10.1.99.26:36997 - 30359 &quot;A IN grafana.com. udp 29 false 512&quot; NOERROR - 0 2.0002791s
[ERROR] plugin/errors: 2 grafana.com. A: read udp 10.1.99.19:37554-&gt;8.8.4.4:53: i/o timeout
[INFO] 10.1.99.32:57927 - 53858 &quot;A IN google.com.mshome.net. udp 39 false 512&quot; NOERROR - 0 2.0001987s
[ERROR] plugin/errors: 2 google.com.mshome.net. A: read udp 10.1.99.19:34079-&gt;8.8.4.4:53: i/o timeout
[INFO] 10.1.99.32:38403 - 36398 &quot;A IN google.com.mshome.net. udp 39 false 512&quot; NOERROR - 0 2.000224s
[ERROR] plugin/errors: 2 google.com.mshome.net. A: read udp 10.1.99.19:59835-&gt;8.8.8.8:53: i/o timeout
[INFO] 10.1.99.26:57447 - 20295 &quot;AAAA IN grafana.com.mshome.net. udp 40 false 512&quot; NOERROR - 0 2.0001892s
[ERROR] plugin/errors: 2 grafana.com.mshome.net. AAAA: read udp 10.1.99.19:51534-&gt;8.8.8.8:53: i/o timeout
[INFO] 10.1.99.26:41052 - 56059 &quot;A IN grafana.com.mshome.net. udp 40 false 512&quot; NOERROR - 0 2.0001879s
[ERROR] plugin/errors: 2 grafana.com.mshome.net. A: read udp 10.1.99.19:47378-&gt;8.8.8.8:53: i/o timeout
[INFO] 10.1.99.26:56748 - 51804 &quot;AAAA IN grafana.com.mshome.net. udp 40 false 512&quot; NOERROR - 0 2.0003226s
[INFO] 10.1.99.26:45442 - 61916 &quot;A IN grafana.com.mshome.net. udp 40 false 512&quot; NOERROR - 0 2.0001922s
[ERROR] plugin/errors: 2 grafana.com.mshome.net. AAAA: read udp 10.1.99.19:35528-&gt;8.8.8.8:53: i/o timeout
[ERROR] plugin/errors: 2 grafana.com.mshome.net. A: read udp 10.1.99.19:53568-&gt;8.8.8.8:53: i/o timeout
</code></pre>
<h3>OS</h3>
<pre class="lang-sh prettyprint-override"><code>pig@pig202:~$ cat /etc/os-release
NAME=&quot;Ubuntu&quot;
VERSION=&quot;20.04 LTS (Focal Fossa)&quot;
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME=&quot;Ubuntu 20.04 LTS&quot;
VERSION_ID=&quot;20.04&quot;
</code></pre>
<p>Tried on Ubuntu 18.04.3 LTS, same issue.</p>
<h3>Other</h3>
<p><code>mshome.net</code> search domain comes from Hyper-V network, I assume. Perhaps this will be of help:</p>
<pre class="lang-sh prettyprint-override"><code>pig@pig202:~$ nmcli device show eth0
GENERAL.DEVICE:                         eth0
GENERAL.TYPE:                           ethernet
GENERAL.HWADDR:                         00:15:5D:88:26:02
GENERAL.MTU:                            1500
GENERAL.STATE:                          100 (connected)
GENERAL.CONNECTION:                     Wired connection 1
GENERAL.CON-PATH:                       /org/freedesktop/NetworkManager/ActiveConnection/1
WIRED-PROPERTIES.CARRIER:               on
IP4.ADDRESS[1]:                         172.19.120.188/28
IP4.GATEWAY:                            172.19.120.177
IP4.ROUTE[1]:                           dst = 0.0.0.0/0, nh = 172.19.120.177, mt = 100
IP4.ROUTE[2]:                           dst = 172.19.120.176/28, nh = 0.0.0.0, mt = 100
IP4.ROUTE[3]:                           dst = 169.254.0.0/16, nh = 0.0.0.0, mt = 1000
IP4.DNS[1]:                             172.19.120.177
IP4.DOMAIN[1]:                          mshome.net
IP6.ADDRESS[1]:                         fe80::6b4a:57e2:5f1b:f739/64
IP6.GATEWAY:                            --
IP6.ROUTE[1]:                           dst = fe80::/64, nh = ::, mt = 100
IP6.ROUTE[2]:                           dst = ff00::/8, nh = ::, mt = 256, table=255
</code></pre>

<p>Finally found the solution which was the combination of two changes. After applying both changes, my pods could finally resolve addresses properly.</p>
<h3>Kubelet configuration</h3>
<p>Based on <a href="https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/#known-issues" rel="noreferrer">known issues</a>, change resolv-conf path for Kubelet to use.</p>
<pre class="lang-sh prettyprint-override"><code># Add resolv-conf flag to Kubelet configuration
echo &quot;--resolv-conf=/run/systemd/resolve/resolv.conf&quot; &gt;&gt; /var/snap/microk8s/current/args/kubelet
# Restart Kubelet
sudo service snap.microk8s.daemon-kubelet restart
</code></pre>
<h3>CoreDNS forward</h3>
<p>Change forward address in CoreDNS config map from default (<code>8.8.8.8 8.8.4.4</code>) to DNS on <code>eth0</code> device.</p>
<pre class="lang-sh prettyprint-override"><code># Dump definition of CoreDNS
microk8s.kubectl get configmap -n kube-system coredns -o yaml &gt; coredns.yaml
</code></pre>
<p>Partial content of coredns.yaml:</p>
<pre class="lang-yaml prettyprint-override"><code> Corefile: |
    .:53 {
        errors
        health {
          lameduck 5s
        }
        ready
        log . {
          class error
        }
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . 8.8.8.8 8.8.4.4
        cache 30
        loop
        reload
        loadbalance
    }

</code></pre>
<p>Fetch DNS:</p>
<pre class="lang-sh prettyprint-override"><code># Fetch eth0 DNS address (this will print 172.19.120.177 in my case)
nmcli dev show 2&gt;/dev/null | grep DNS | sed 's/^.*:\s*//'
</code></pre>
<p>Change the following line and save:</p>
<pre class="lang-sh prettyprint-override"><code>        forward . 8.8.8.8 8.8.4.4 # From this
        forward . 172.19.120.177 # To this (your DNS will probably be different)
</code></pre>
<p>Finally apply to change CoreDNS forwarding:</p>
<pre class="lang-sh prettyprint-override"><code>microk8s.kubectl apply -f coredns.yaml
</code></pre>
