<p>My K8 DNS isn't resolving, thus I did follow the debugging steps as <a href="https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/" rel="nofollow noreferrer">mentioned here</a>. As I am new to K8, can someone point me to the issue I am facing? I cant extract any useful information out of the debugging steps.</p>
<blockquote>
<p>cat /etc/os-release</p>
</blockquote>
<pre><code>NAME=&quot;Ubuntu&quot;
VERSION=&quot;20.04.2 LTS (Focal Fossa)&quot;
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME=&quot;Ubuntu 20.04.2 LTS&quot;
</code></pre>
<blockquote>
<p>kubectl version</p>
</blockquote>
<pre><code>Client Version: version.Info{Major:&quot;1&quot;, Minor:&quot;20&quot;, 
GitVersion:&quot;v1.20.5&quot;, GitCommit:&quot;6b1d87acf3c8253c123756b9e61dac642678305f&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2021-03-18T01:10:43Z&quot;, GoVersion:&quot;go1.15.8&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}
Server Version: version.Info{Major:&quot;1&quot;, Minor:&quot;20&quot;, GitVersion:&quot;v1.20.5&quot;, GitCommit:&quot;6b1d87acf3c8253c123756b9e61dac642678305f&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2021-03-18T01:02:01Z&quot;, GoVersion:&quot;go1.15.8&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}
</code></pre>
<blockquote>
<p>kubectl get namespace</p>
</blockquote>
<pre><code>NAME                   STATUS   AGE
default                Active   7d4h
</code></pre>
<blockquote>
<p>kubectl get pods dnsutils</p>
</blockquote>
<pre><code>NAME       READY   STATUS    RESTARTS   AGE
dnsutils   1/1     Running   18         18h
</code></pre>
<blockquote>
<p>kubectl exec -i -t dnsutils -- nslookup kubernetes.default</p>
</blockquote>
<pre><code>;; connection timed out; no servers could be reached
</code></pre>
<blockquote>
<p>kubectl exec -ti dnsutils -- cat /etc/resolv.conf</p>
</blockquote>
<pre><code>nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local 
cluster.local
options ndots:5
</code></pre>
<blockquote>
<p>kubectl get pods --namespace=kube-system -l k8s-app=kube-dns</p>
</blockquote>
<pre><code>NAME                      READY   STATUS    RESTARTS   
AGE
coredns-74ff55c5b-6vsml   1/1     Running   12  7d4h
coredns-74ff55c5b-mww7g   1/1     Running   12  7d4h
</code></pre>
<blockquote>
<p>kubectl logs --namespace=kube-system -l k8s-app=kube-dns</p>
</blockquote>
<pre><code>.:53
[INFO] plugin/reload: Running configuration MD5 = db32ca3650231d74073ff4cf814959a7
CoreDNS-1.7.0
linux/amd64, go1.14.4, f59c03d
[INFO] Reloading
[INFO] plugin/health: Going into lameduck mode for 5s
[INFO] plugin/reload: Running configuration MD5 = 3d3f6363f05ccd60e0f885f0eca6c5ff
[INFO] Reloading complete
[INFO] 10.244.0.1:16732 - 59651 &quot;HINFO IN 6307445054232439722.7934820194057826263. udp 57 false 512&quot; NXDOMAIN qr,rd,ra 132 0.006053527s
[INFO] 127.0.0.1:58672 - 59651 &quot;HINFO IN 6307445054232439722.7934820194057826263. udp 57 false 512&quot; NXDOMAIN qr,rd,ra 132 0.00658948s
[INFO] plugin/reload: Running configuration MD5 = db32ca3650231d74073ff4cf814959a7
CoreDNS-1.7.0
linux/amd64, go1.14.4, f59c03d
[INFO] Reloading
[INFO] plugin/health: Going into lameduck mode for 5s
[INFO] plugin/reload: Running configuration MD5 = 3d3f6363f05ccd60e0f885f0eca6c5ff
[INFO] Reloading complete
[INFO] 10.244.0.62:56364 - 32900 &quot;HINFO IN 2808379183970575835.6786373795048579500. udp 57 false 512&quot; NXDOMAIN qr,rd,ra 132 0.004922932s
[INFO] 127.0.0.1:48277 - 32900 &quot;HINFO IN 2808379183970575835.6786373795048579500. udp 57 false 512&quot; NXDOMAIN qr,rd,ra 132 0.007889024s
[INFO] 10.244.0.62:49106 - 59651 &quot;HINFO IN 6307445054232439722.7934820194057826263. udp 57 false 512&quot; NXDOMAIN qr,rd,ra 132 0.005058199s
</code></pre>
<blockquote>
<p>kubectl get svc --namespace=kube-system</p>
</blockquote>
<pre><code>NAME                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                  AGE
kube-dns              ClusterIP   10.96.0.10      &lt;none&gt;        53/UDP,53/TCP,9153/TCP   7d4h
monitoring-influxdb   ClusterIP   10.102.51.183   &lt;none&gt;        8086/TCP                 4d21h
</code></pre>
<blockquote>
<p>kubectl get endpoints kube-dns --namespace=kube-system</p>
</blockquote>
<pre><code>NAME       ENDPOINTS                                                  AGE
kube-dns   10.244.0.45:53,10.244.0.47:53,10.244.0.45:53 + 3 more...   7d4h
</code></pre>
<blockquote>
<p>cat /run/systemd/resolve/resolv.conf</p>
</blockquote>
<pre><code>nameserver 8.8.8.8
nameserver 2001:4860:4860::8888
</code></pre>
<blockquote>
<p>cat /etc/systemd/resolved.conf</p>
</blockquote>
<pre><code>[Resolve]
DNS=8.8.8.8 2001:4860:4860::8888
</code></pre>
<blockquote>
<p>cat /etc/resolv.conf</p>
</blockquote>
<pre><code>nameserver 127.0.0.53
options edns0 trust-ad
</code></pre>
<p>It is kinda odd, that both resolv.conf have different values. Also, I have no clue (if I would have to set the DNS IP manually) which IP to choose.</p>
<blockquote>
<p>kubeadm config view</p>
</blockquote>
<pre><code>apiServer:
  extraArgs:
    authorization-mode: Node,RBAC
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: k8s.gcr.io
kind: ClusterConfiguration
kubernetesVersion: v1.20.5
networking:
  dnsDomain: cluster.local
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.96.0.0/12
scheduler: {}
</code></pre>
<p><strong>Update</strong></p>
<p>The dnsutils assigned pod's IP is 10.244.2.20 and not reachable from the single k8 master node.</p>
<pre><code>ping 10.244.2.20
</code></pre>

<p>There were several issues with my configuration. First off: I did use an incompatible docker version (20.10.5) which isn't supported yet. Hence, I don't know whether this issue also arises when using a supported docker version. However, even with this incompatible docker version, I was able to fix the issue with following steps:</p>
<p><strong>1. DNS misconfiguration</strong></p>
<p>I don't know who/what will set the <code>resolved.conf</code>'s DNS entries, but my entry was clearly wrong. First, we need to obtain the K8's DNS Cluster-IP Address:</p>
<blockquote>
<p>kubectl get services --all-namespaces -o wide</p>
</blockquote>
<p>You will receive all services within all namespaces, including the <code>kube-dns</code> Cluster-IP. In my case It looks like following</p>
<pre><code>NAMESPACE              NAME                                    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                  AGE   SELECTOR
kube-system            kube-dns                                ClusterIP   10.96.0.10       &lt;none&gt;        53/UDP,53/TCP,9153/TCP   11d   k8s-app=kube-dns
kube-system            monitoring-influxdb                     ClusterIP   10.102.51.183    &lt;none&gt;        8086/TCP                 9d    k8s-app=influxdb
kubernetes-dashboard   dashboard-metrics-scraper               ClusterIP   10.110.126.218   &lt;none&gt;        8000/TCP                 11d   k8s-app=dashboard-metrics-scraper
kubernetes-dashboard   kubernetes-dashboard                    ClusterIP   10.98.164.199    &lt;none&gt;        443/TCP                  11d   k8s-app=kubernetes-dashboard
</code></pre>
<p>Use that DNS within your <code>resolved.conf</code> file. Where that file is located, depends on your OS. In my case (Ubuntu 20.04) <code>/etc/systemd/resolved.conf</code>.</p>
<blockquote>
<p>nano /etc/systemd/resolved.conf</p>
</blockquote>
<pre><code>[Resolve]
DNS=10.96.0.10 8.8.8.8 2001:4860:4860::8888
</code></pre>
<p><strong>2. Re-Join all nodes</strong></p>
<p>I did use UFW next to IPTables, which was somehow messing with the configuration. Hence, I did <a href="https://stackoverflow.com/a/60070134/2934290">remove all nodes</a>, installed a fresh OS and re-joined the cluster; without activating UFW.</p>
<p><strong>3. Forward packet policy</strong></p>
<p>In some versions <a href="https://github.com/flannel-io/flannel/issues/799#issuecomment-325581861" rel="nofollow noreferrer">docker modifies the iptables</a>, such that packets will be dropped in packet-forward scenarios. Override this behaviour on all nodes with:</p>
<pre><code>iptables -P FORWARD ACCEPT
</code></pre>
<p>Just to be sure, also enable ipv4 forwarding with:</p>
<pre><code>echo &quot;net.ipv4.ip_forward=1&quot; &gt;&gt; /etc/sysctl.conf
</code></pre>
