<p>I installed a clean K8s cluster in virtual machines (Debian 10). After the installation and the integration into my landscape, I checked the connectivity inside my testing alpine image. As result the connection of outgoing traffic not working and no information was inside the coreDNS log. I used the workaround on my build image to overwrite my /etc/resolv.conf and replace the DNS entries (e.g. set 1.1.1.1 as Nameserver). After that temporary &quot;hack&quot; the connection to the internet works perfectly. But the workaround is not a long term solution and I want to use the official way. Inside the documentation of K8s coreDNS, I found the forward section and I interpret the flag like an option, to forward the inquiry to the predefined local resolver. I think the forwarding to the local resolv.conf and the resolve process works not correctly. Can anyone help me to solve that issue?</p>
<p><strong>Basic setup:</strong></p>
<ul>
<li>K8s version: 1.19.0</li>
<li>K8s setup: 1 master + 2 worker nodes</li>
<li>Based on: Debian 10 VM's</li>
<li>CNI: Flannel</li>
</ul>
<p><strong>Status of CoreDNS Pods</strong></p>
<pre><code>kube-system            coredns-xxxx 1/1     Running   1          26h
kube-system            coredns-yyyy 1/1     Running   1          26h
</code></pre>
<p><strong>CoreDNS Log:</strong></p>
<pre><code>.:53
[INFO] plugin/reload: Running configuration MD5 = 4e235fcc3696966e76816bcd9034ebc7
CoreDNS-1.6.7
</code></pre>
<p><strong>CoreDNS config:</strong></p>
<pre><code>apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
kind: ConfigMap
metadata:
  creationTimestamp: &quot;&quot;
  name: coredns
  namespace: kube-system
  resourceVersion: &quot;219&quot;
  selfLink: /api/v1/namespaces/kube-system/configmaps/coredns
  uid: xxx
</code></pre>
<p><strong>Ouput alpine image:</strong></p>
<pre><code>/ # nslookup -debug google.de
;; connection timed out; no servers could be reached
</code></pre>
<p><strong>Output of pods resolv.conf</strong></p>
<pre><code>/ # cat /etc/resolv.conf 
nameserver 10.96.0.10
search development.svc.cluster.local svc.cluster.local cluster.local invalid
options ndots:5
</code></pre>
<p><strong>Output of host resolv.conf</strong></p>
<pre><code>cat /etc/resolv.conf 
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 213.136.95.11
nameserver 213.136.95.10
search invalid
</code></pre>
<p><strong>Output of host /run/flannel/subnet.env</strong></p>
<pre><code>cat /run/flannel/subnet.env
FLANNEL_NETWORK=10.244.0.0/16
FLANNEL_SUBNET=10.244.0.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true
</code></pre>
<p><strong>Output of <code>kubectl get pods -n kube-system -o wide</code></strong></p>
<pre><code>coredns-54694b8f47-4sm4t                 1/1     Running   0          14d   10.244.1.48    xxx3-node-1   &lt;none&gt;           &lt;none&gt;
coredns-54694b8f47-6c7zh                 1/1     Running   0          14d   10.244.0.43    xxx2-master   &lt;none&gt;           &lt;none&gt;
coredns-54694b8f47-lcthf                 1/1     Running   0          14d   10.244.2.88    xxx4-node-2   &lt;none&gt;           &lt;none&gt;
etcd-xxx2-master                      1/1     Running   7          27d   xxx.xx.xx.xxx   xxx2-master   &lt;none&gt;           &lt;none&gt;
kube-apiserver-xxx2-master            1/1     Running   7          27d   xxx.xx.xx.xxx   xxx2-master   &lt;none&gt;           &lt;none&gt;
kube-controller-manager-xxx2-master   1/1     Running   7          27d   xxx.xx.xx.xxx   xxx2-master   &lt;none&gt;           &lt;none&gt;
kube-flannel-ds-amd64-4w8zl              1/1     Running   8          28d   xxx.xx.xx.xxx   xxx2-master   &lt;none&gt;           &lt;none&gt;
kube-flannel-ds-amd64-w7m44              1/1     Running   7          28d   xxx.xx.xx.xxx   xxx3-node-1   &lt;none&gt;           &lt;none&gt;
kube-flannel-ds-amd64-xztqm              1/1     Running   6          28d   xxx.xx.xx.xxx   xxx4-node-2   &lt;none&gt;           &lt;none&gt;
kube-proxy-dfs85                         1/1     Running   4          28d   xxx.xx.xx.xxx   xxx4-node-2   &lt;none&gt;           &lt;none&gt;
kube-proxy-m4hl2                         1/1     Running   4          28d   xxx.xx.xx.xxx   xxx3-node-1   &lt;none&gt;           &lt;none&gt;
kube-proxy-s7p4s                         1/1     Running   8          28d   xxx.xx.xx.xxx   xxx2-master   &lt;none&gt;           &lt;none&gt;
kube-scheduler-xxx2-master            1/1     Running   7          27d   xxx.xx.xx.xxx   xxx2-master   &lt;none&gt;           &lt;none&gt;
</code></pre>

<p><strong>Problem:</strong></p>
<p>The (two) coreDNS pods were only deployed on the master node. You can check the settings with this command.</p>
<pre><code>kubectl get pods -n kube-system -o wide | grep coredns
</code></pre>
<p><strong>Solution:</strong></p>
<p>I could solve the problem by scaling up the coreDNS pods and edit the deployment configuration. The following commands must be executed.</p>
<ol>
<li><code>kubectl edit deployment coredns -n kube-system</code></li>
<li><code>Set replicas value to node quantity e.g. 3</code></li>
<li><code>kubectl patch deployment coredns -n kube-system -p &quot;{\&quot;spec\&quot;:{\&quot;template\&quot;:{\&quot;metadata\&quot;:{\&quot;annotations\&quot;:{\&quot;force-update/updated-at\&quot;:\&quot;$(date +%s)\&quot;}}}}}&quot;</code></li>
<li><code>kubectl get pods -n kube-system -o wide | grep coredns</code></li>
</ol>
<p><strong>Source</strong></p>
<ul>
<li><a href="https://blog.dbi-services.com/kubernetes-dns-resolution-using-coredns-force-update-deployment/" rel="nofollow noreferrer">https://blog.dbi-services.com/kubernetes-dns-resolution-using-coredns-force-update-deployment/</a></li>
</ul>
<p><strong>Hint</strong></p>
<p>If you still have a problem with your coreDNS and your DNS resolution works sporadically, take a look at this <a href="https://serverfault.com/questions/1033711/k8s-no-dns-resolution-on-all-used-test-images">post</a>.</p>
