<p>I have two kubernetes pods running via Rancher:</p>
<p>#1 - busybox
#2 - dnsutils</p>
<p>From the pod #1:</p>
<pre><code>/ # cat /etc/resolv.conf 
nameserver 10.43.0.10
search testspace.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
</code></pre>
<p>and then</p>
<pre><code>/ # nslookup kubernetes.default
Server:    10.43.0.10
Address 1: 10.43.0.10 kube-dns.kube-system.svc.cluster.local

nslookup: can't resolve 'kubernetes.default'
/ # nslookup kubernetes.default
Server:    10.43.0.10
Address 1: 10.43.0.10 kube-dns.kube-system.svc.cluster.local

nslookup: can't resolve 'kubernetes.default'
/ # nslookup kubernetes.default
Server:    10.43.0.10
Address 1: 10.43.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes.default
Address 1: 10.43.0.1 kubernetes.default.svc.cluster.local
</code></pre>
<p>so sometimes it works but mostly not.</p>
<p>then from the pod #2:</p>
<pre><code>nameserver 10.43.0.10
search testspace.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
</code></pre>
<p>and then:</p>
<pre><code>/ # nslookup kubernetes.default
;; connection timed out; no servers could be reached

/ # nslookup kubernetes.default
;; connection timed out; no servers could be reached

/ # nslookup kubernetes.default
Server:         10.43.0.10
Address:        10.43.0.10#53

Name:   kubernetes.default.svc.cluster.local
Address: 10.43.0.1
;; connection timed out; no servers could be reached
</code></pre>
<p>so it mostly doesn't work.</p>
<p>The same problem is when I try to reach any external hostname.</p>
<p>Also tried to troubleshoot based on article from <a href="https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/" rel="nofollow noreferrer">here</a></p>
<p>ConfigMap:</p>
<pre><code>kubectl -n kube-system edit configmap coredns

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
  Corefile: |
    .:53 {
        log
        errors
        health {
          lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . &quot;/etc/resolv.conf&quot;
        cache 30
        loop
        reload
        loadbalance
    }
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {&quot;apiVersion&quot;:&quot;v1&quot;,&quot;data&quot;:{&quot;Corefile&quot;:&quot;.:53 {\n    errors\n    health {\n      lameduck 5s\n    }\n    ready\n    kubernetes cluster.local in-addr.arpa ip6.arpa {\n      pods insecure\n      fallthrough in-addr.arpa ip6.arpa\n    }\n    prometheus :9153\n    forward . \&quot;/etc/resolv.conf\&quot;\n    cache 30\n    loop\n    reload\n    loadbalance\n}\n&quot;},&quot;kind&quot;:&quot;ConfigMap&quot;,&quot;metadata&quot;:{&quot;annotations&quot;:{},&quot;name&quot;:&quot;coredns&quot;,&quot;namespace&quot;:&quot;kube-system&quot;}}
  creationTimestamp: &quot;2020-08-07T19:28:25Z&quot;
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data:
        .: {}
        f:Corefile: {}
      f:metadata:
        f:annotations:
          .: {}
          f:kubectl.kubernetes.io/last-applied-configuration: {}
    manager: kubectl
    operation: Update
    time: &quot;2020-08-24T19:22:17Z&quot;
  name: coredns
  namespace: kube-system
  resourceVersion: &quot;4118524&quot;
  selfLink: /api/v1/namespaces/kube-system/configmaps/coredns
  uid: 1f3615b0-9349-4bc5-990b-7fed31879fa2
~                                          
</code></pre>
<p>Any thought on that?</p>

<p>It came up that <code>kube-dns</code> service was not able to get CoreDNS pods</p>
<pre><code>&gt; kubectl get svc -o wide --namespace=kube-system

NAME             TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                  AGE   SELECTOR
kube-dns         ClusterIP   10.43.0.10     &lt;none&gt;        53/UDP,53/TCP,9153/TCP   24d   k8s-app=kube-dns
</code></pre>
<p>when CoreDNS from one node called directly to the pod was able</p>
<pre><code>/ # nslookup google.com 10.42.1.18 
Server:         10.42.1.18
Address:        10.42.1.18#53

Non-authoritative answer:
Name:   google.com
Address: 172.217.10.110
Name:   google.com
Address: 2607:f8b0:4006:802::200e
</code></pre>
<p>and another node was not:</p>
<pre><code>/ # nslookup google.com 10.42.2.37
;; connection timed out; no servers could be reached
</code></pre>
<p>which may created problem for the <code>kube-dns</code> service.</p>
<p>In this case I have decided to rebuild that problematic node and problem went away.</p>
