<p>I have deployed a Kubernetes cluster composed of a master and two workers using <code>kubeadm</code> and the Flannel network driver (So I passed the <code>--pod-network-cidr=10.244.0.0/16</code> flag to <code>kubeadm init</code>).</p>

<p>Those nodes are communicating together using a VPN so that:</p>

<ul>
<li>Master node IP address is 10.0.0.170</li>
<li>Worker 1 IP address is 10.0.0.247</li>
<li>Worker 2 IP address is 10.0.0.35</li>
</ul>

<p>When I create a new pod and I try to ping google I have the following error:</p>

<pre><code>/ # ping google.com
ping: bad address 'google.com'
</code></pre>

<p>I followed the instructions from the <a href="https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/" rel="noreferrer">Kubernetes DNS debugging resolution</a> documentation page:</p>

<pre><code>$ kubectl exec -ti busybox -- nslookup kubernetes.default
Server:    10.96.0.10
Address 1: 10.96.0.10

nslookup: can't resolve 'kubernetes.default'
command terminated with exit code 1
</code></pre>

<h3>Check the local DNS configuration first</h3>

<pre><code>$ kubectl exec busybox cat /etc/resolv.conf
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local invalid
options ndots:5
</code></pre>

<h3>Check if the DNS pod is running</h3>

<pre><code>$ kubectl get pods --namespace=kube-system -l k8s-app=kube-dns
NAME                       READY   STATUS    RESTARTS   AGE
coredns-5c98db65d4-cqzb7   1/1     Running   0          7d18h
coredns-5c98db65d4-xc5d7   1/1     Running   0          7d18h
</code></pre>

<h3>Check for Errors in the DNS pod</h3>

<pre><code>$ for p in $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name); do kubectl logs --namespace=kube-system $p; done
.:53
2019-10-28T13:40:41.834Z [INFO] CoreDNS-1.3.1
2019-10-28T13:40:41.834Z [INFO] linux/amd64, go1.11.4, 6b56a9c
CoreDNS-1.3.1
linux/amd64, go1.11.4, 6b56a9c
2019-10-28T13:40:41.834Z [INFO] plugin/reload: Running configuration MD5 = 5d5369fbc12f985709b924e721217843
.:53
2019-10-28T13:40:42.870Z [INFO] CoreDNS-1.3.1
2019-10-28T13:40:42.870Z [INFO] linux/amd64, go1.11.4, 6b56a9c
CoreDNS-1.3.1
linux/amd64, go1.11.4, 6b56a9c
2019-10-28T13:40:42.870Z [INFO] plugin/reload: Running configuration MD5 = 5d5369fbc12f985709b924e721217843
</code></pre>

<h3>Is DNS service up?</h3>

<pre><code>$ kubectl get svc --namespace=kube-system
NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
kube-dns   ClusterIP   10.96.0.10   &lt;none&gt;        53/UDP,53/TCP,9153/TCP   7d18h
</code></pre>

<h3>Are DNS endpoints exposed?</h3>

<pre><code>$ kubectl get ep kube-dns --namespace=kube-system
NAME       ENDPOINTS                                               AGE
kube-dns   10.244.0.3:53,10.244.0.4:53,10.244.0.3:53 + 3 more...   7d18h
</code></pre>

<h3>Are DNS queries being received/processed?</h3>

<p>I made the update to the coredns ConfigMap, ran again the <code>nslookup kubernetes.default</code> command, and here is the result:</p>

<pre><code>$ for p in $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name); do kubectl logs --namespace=kube-system $p; done
.:53
2019-10-28T13:40:41.834Z [INFO] CoreDNS-1.3.1
2019-10-28T13:40:41.834Z [INFO] linux/amd64, go1.11.4, 6b56a9c
CoreDNS-1.3.1
linux/amd64, go1.11.4, 6b56a9c
2019-10-28T13:40:41.834Z [INFO] plugin/reload: Running configuration MD5 = 5d5369fbc12f985709b924e721217843
[INFO] Reloading
2019-11-05T08:12:12.511Z [INFO] plugin/reload: Running configuration MD5 = 906291470f7b1db8bef629bdd0056cad
[INFO] Reloading complete
2019-11-05T08:12:12.608Z [INFO] 127.0.0.1:55754 - 7434 "HINFO IN 4808438627636259158.5471394156194192600. udp 57 false 512" NXDOMAIN qr,rd,ra 132 0.095189791s
.:53
2019-10-28T13:40:42.870Z [INFO] CoreDNS-1.3.1
2019-10-28T13:40:42.870Z [INFO] linux/amd64, go1.11.4, 6b56a9c
CoreDNS-1.3.1
linux/amd64, go1.11.4, 6b56a9c
2019-10-28T13:40:42.870Z [INFO] plugin/reload: Running configuration MD5 = 5d5369fbc12f985709b924e721217843
[INFO] Reloading
2019-11-05T08:12:47.988Z [INFO] plugin/reload: Running configuration MD5 = 906291470f7b1db8bef629bdd0056cad
[INFO] Reloading complete
2019-11-05T08:12:48.004Z [INFO] 127.0.0.1:51911 - 60104 "HINFO IN 4077052818408395245.3902243105088660270. udp 57 false 512" NXDOMAIN qr,rd,ra 132 0.016522153s
</code></pre>

<p>So it seems that DNS pods are receiving the requests.</p>

<h2>But I had this error already!</h2>

<p>That error happened to me the first time I deployed the cluster.</p>

<p>At that time, I noticed that <code>kubectl get nodes -o wide</code> was showing the workers node public IP address as "INTERNAL-IP" instead of the private one.</p>

<p>Looking further I found out that on the worker nodes, kubelet was missing the <code>--node-ip</code> flag, so I've added it and restarted Kubelet and the issue was gone. I then concluded that missing flag was the reason, but it seems to not be the case as the <code>kubectl get nodes -o wide</code> command shows the internal IP addresses as "INTERNAL-IP" for the workers.</p>

<h2>And now</h2>

<p>The DNS server IP address 10.96.0.10 looks wrong to me, and I can't ping it from the pod. The DNS pods have the IP addresses 10.244.0.3 and 10.244.0.4 which I can't ping too.</p>

<p>I just tried to delete the coredns pods, so that they are scheduled again, and now their IP addresses changed, I can ping them from the pod and the <code>kubectl exec -ti busybox -- nslookup kubernetes.default</code> works:</p>

<pre><code>$ kubectl exec -ti busybox -- nslookup kubernetes.default
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes.default
Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local
</code></pre>

<p>But the <code>resolv.conf</code> file still has the "invalid" inside:</p>

<pre><code>$ kubectl exec busybox cat /etc/resolv.conf
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local invalid
options ndots:5
</code></pre>

<ul>
<li>Can anyone explain me what happened please?</li>
<li>And how can I solved this "invalid" from the <code>resolv.conf</code> file?</li>
</ul>

<p>As configured in <a href="https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/#coredns" rel="noreferrer">CoreDNS ConfigMap</a> default upstream nameservers are inherited from node, that is everything outside the cluster domain (.cluster.local)</p>

<p>So "invalid" is an entry copied from Node's <code>/etc/resolv.conf</code> file during Pod creation.</p>

<p>If you would manually modify <code>/etc/resolv.conf</code> on you Node, every Pod with <code>dnsPolicy: ClusterFirst</code> will inherit <code>/etc/resolv.conf</code> with this modification.</p>

<p>So, after adding <code>--node-ip</code> flag to kubelet and restarting CoreDNS Pods you should re-deploy your busybox Pod so it can inherit <code>/etc/resolv.conf</code> from the Node.</p>
