<p>I have a local cluster with minikube 1.6.2 running. </p>

<p>All my pods are OK, I checked the logs individually, but I have 2 db, influx and postgres, are not accesible anymore from any url outside namespace.</p>

<p>I logged into both pods, and I can confirm that each db is OK, has data, and I can connect manually with my user / pass.</p>

<p>Let's take the case of influx.</p>

<pre><code>kubectl exec -it -n influx blockchain-influxdb-local-fb745b98c-vbghp -- influx -username='myuser' -password="mypass" -database="mydb" -precision=rfc3339 -execute "show measurements"
</code></pre>

<p>gives me 4 measurements, so no pb.</p>

<p>but when I try to connect influx from the same namespace with his local dns, I get a timeout.</p>

<pre><code>➜ ~ kubectl get svc -n influx 
NAME                        TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)     AGE
blockchain-influxdb-local   ClusterIP   10.96.175.62   &lt;none&gt;        8086/TCP    19m


➜ ~ kubectl get deployments -n influx
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE
blockchain-influxdb-local   1/1     1            1           20m


➜  ~ kubectl get po -n influx
NAME                                        READY   STATUS    RESTARTS   AGE
blockchain-influxdb-local-fb745b98c-vbghp   1/1     Running   0          21m
measures-api-local-8667bb496f-4wp8d         1/1     Running   0          21m
</code></pre>

<p>Case where it works:</p>

<p>From a pod inside the same namespace:</p>

<pre><code>curl --verbose -G 'http://blockchain-influxdb-local:8086/query?db=mydb&amp;pretty=true' --data-urlencode 'u=myuser' --data-urlencode 'p=mypass' --data-urlencode 'precision=rfc3339' --data-urlencode 'q=show measurements'
</code></pre>

<p>From a pod in another namespace (same namespace), with pod IP</p>

<pre><code>curl --verbose -G '172.17.0.5:8086/query?db=mydb&amp;pretty=true' --data-urlencode 'u=myuser' --data-urlencode 'p=mypass' --data-urlencode 'precision=rfc3339' --data-urlencode 'q=show measurements'
</code></pre>

<p>From a pod in another namespace (same namespace), with service IP</p>

<pre><code>curl --verbose -G '10.96.175.62:8086/query?db=mydb&amp;pretty=true' --data-urlencode 'u=myuser' --data-urlencode 'p=mypass' --data-urlencode 'precision=rfc3339' --data-urlencode 'q=show measurements'
</code></pre>

<p>But when I use local dns from outside namespace, it won't work, I get a timeout from CURL:</p>

<pre><code> curl --verbose -G 'blockchain-influxdb-local.influx.svc.cluster.local:8086/query?db=mydb&amp;pretty=true' --data-urlencode 'u=myuser' --data-urlencode 'p=mypass' --data-urlencode 'precision=rfc3339' --data-urlencode 'q=show measurements'
</code></pre>

<p>I followed those debug step to ensure DNS is working, and had no problem, everything works.</p>

<p><a href="https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/" rel="nofollow noreferrer">https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/</a></p>

<p>Inside same pod, when I ping this url, I get:</p>

<pre><code>root@metadata-api-local-8b4b7846b-zllb8:/go/src/gitlab.com/company/metadata_api# ping blockchain-influxdb-local.influx.svc.cluster.local

PING nc-ass-vip.sdv.fr (212.95.74.75) 56(84) bytes of data.
--- nc-ass-vip.sdv.fr ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 47ms
</code></pre>

<p>I don't know why is it making a reference to <code>nc-ass-vip.sdv.fr</code></p>

<p>I also tried to remove local cluster and redeploy it, also tried to update minikube to latest version (1.8.2), nothing worked.</p>

<p>I don't know what else to do...</p>

<p>Has anyone an idea ? I was working well for monthes, don't really know what happened. :(</p>

<p>In response to @Arghya Sadhu, I post the file /etc/resolv.conf from the Influx pod:</p>

<pre><code>nameserver 10.96.0.10
search influx.svc.cluster.local svc.cluster.local cluster.local numericable.fr
options ndots:5
</code></pre>

<p><code>kubectl edit cm coredns -n kube-system</code></p>

<pre><code># Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
kind: ConfigMap
metadata:
  creationTimestamp: "2020-03-19T10:59:28Z"
  name: coredns
  namespace: kube-system
  resourceVersion: "176"
  selfLink: /api/v1/namespaces/kube-system/configmaps/coredns
  uid: 0797c1a9-e9db-4b4c-bc8d-4c7ecca24968
</code></pre>

<p>EDIT: </p>

<pre><code>kubectl exec -ti dnsutils -- nslookup blockchain-influxdb-local.influx.svc.cluster.local                                         
Server:     10.96.0.10
Address:    10.96.0.10#53

Non-authoritative answer:
blockchain-influxdb-local.influx.svc.cluster.local.numericable.fr   canonical name = nc-ass-vip.sdv.fr.
Name:   nc-ass-vip.sdv.fr
Address: 212.95.74.75
</code></pre>

<p>After digging into a few possibilities we came across the output for the following commands: </p>

<pre><code>$ kubectl run dnsutils -it --rm=true --restart=Never --image=tutum/dnsutils -- nslookup -debug blockchain-influxdb-local.influx
</code></pre>

<pre><code>$ kubectl run dnsutils -it --rm=true --restart=Never --image=tutum/dnsutils -- nslookup -debug blockchain-influxdb-local.influx.svc.cluster.local
</code></pre>

<pre><code>$ kubectl run dnsutils -it --rm=true --restart=Never --image=tutum/dnsutils -- nslookup -debug blockchain-influxdb-local.influx.svc.cluster.local.
</code></pre>

<p>Output for these commands <a href="https://gist.github.com/xoco70/ea6a99ce7acb31cb06c78e7bfffeaf78" rel="nofollow noreferrer">here</a> (adding to the end of this answer for future reference in case of link doesn't work).</p>

<p>Reviewing this output we can see that no matter what <code>numericable.fr</code> is always giving positive answer to dns queries.  </p>

<p>To avoid this situation you can change ndots entry to 1 or even 0 in your pods. </p>

<pre><code>nameserver 10.96.0.10
search influx.svc.cluster.local svc.cluster.local cluster.local numericable.fr
options ndots:0
</code></pre>

<p>From man pages we have: </p>

<blockquote>
  <p>ndots:n
                Sets a threshold for the number of dots which must appear
                in a name given to res_query(3) (see resolver(3))  before
                an  initial absolute query will be made.  The default for
                n is 1, meaning that if there are any dots in a name, the
                name  will  be tried first as an absolute name before any
                search list elements are appended to it.  The  value  for
                this option is silently capped to 15.</p>
</blockquote>

<p>A more effective and long term solution is to add this entry in the pod/statefulset/deployment manifest as in this example: </p>

<pre><code>apiVersion: v1
kind: Pod
metadata:
  namespace: default
  name: dns-example
spec:
  containers:
    - name: test
      image: nginx
  dnsConfig:
    options:
      - name: ndots
        value: "0"
</code></pre>

<p>Output from commands referenced for future reference: </p>

<pre><code>➜  ~ kubectl run dnsutils -it --rm=true --restart=Never --image=tutum/dnsutils -- nslookup -debug blockchain-influxdb-local.influx 

Server:     10.96.0.10
Address:    10.96.0.10#53

------------
    QUESTIONS:
    blockchain-influxdb-local.influx.default.svc.cluster.local, type = A, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    -&gt;  cluster.local
    origin = ns.dns.cluster.local
    mail addr = hostmaster.cluster.local
    serial = 1584628757
    refresh = 7200
    retry = 1800
    expire = 86400
    minimum = 30
    ttl = 10
    ADDITIONAL RECORDS:
------------
** server can't find blockchain-influxdb-local.influx.default.svc.cluster.local: NXDOMAIN
Server:     10.96.0.10
Address:    10.96.0.10#53

------------
    QUESTIONS:
    blockchain-influxdb-local.influx.svc.cluster.local, type = A, class = IN
    ANSWERS:
    -&gt;  blockchain-influxdb-local.influx.svc.cluster.local
    internet address = 10.96.72.6
    ttl = 10
    AUTHORITY RECORDS:
    ADDITIONAL RECORDS:
------------
Name:   blockchain-influxdb-local.influx.svc.cluster.local
Address: 10.96.72.6

pod "dnsutils" deleted
pod default/dnsutils terminated (Error)
➜  ~ kubectl run dnsutils -it --rm=true --restart=Never --image=tutum/dnsutils -- nslookup -debug blockchain-influxdb-local.influx.svc.cluster.local 

Server:     10.96.0.10
Address:    10.96.0.10#53

------------
    QUESTIONS:
    blockchain-influxdb-local.influx.svc.cluster.local.default.svc.cluster.local, type = A, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    -&gt;  cluster.local
    origin = ns.dns.cluster.local
    mail addr = hostmaster.cluster.local
    serial = 1584628757
    refresh = 7200
    retry = 1800
    expire = 86400
    minimum = 30
    ttl = 30
    ADDITIONAL RECORDS:
------------
** server can't find blockchain-influxdb-local.influx.svc.cluster.local.default.svc.cluster.local: NXDOMAIN
Server:     10.96.0.10
Address:    10.96.0.10#53

------------
    QUESTIONS:
    blockchain-influxdb-local.influx.svc.cluster.local.svc.cluster.local, type = A, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    -&gt;  cluster.local
    origin = ns.dns.cluster.local
    mail addr = hostmaster.cluster.local
    serial = 1584628757
    refresh = 7200
    retry = 1800
    expire = 86400
    minimum = 30
    ttl = 30
    ADDITIONAL RECORDS:
------------
** server can't find blockchain-influxdb-local.influx.svc.cluster.local.svc.cluster.local: NXDOMAIN
Server:     10.96.0.10
Address:    10.96.0.10#53

------------
    QUESTIONS:
    blockchain-influxdb-local.influx.svc.cluster.local.cluster.local, type = A, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    -&gt;  cluster.local
    origin = ns.dns.cluster.local
    mail addr = hostmaster.cluster.local
    serial = 1584628757
    refresh = 7200
    retry = 1800
    expire = 86400
    minimum = 30
    ttl = 30
    ADDITIONAL RECORDS:
------------
** server can't find blockchain-influxdb-local.influx.svc.cluster.local.cluster.local: NXDOMAIN
Server:     10.96.0.10
Address:    10.96.0.10#53

------------
    QUESTIONS:
    blockchain-influxdb-local.influx.svc.cluster.local.numericable.fr, type = A, class = IN
    ANSWERS:
    -&gt;  blockchain-influxdb-local.influx.svc.cluster.local.numericable.fr
    canonical name = nc-ass-vip.sdv.fr.
    ttl = 30
    -&gt;  nc-ass-vip.sdv.fr
    internet address = 212.95.74.75
    ttl = 30
    AUTHORITY RECORDS:
    ADDITIONAL RECORDS:
------------
Non-authoritative answer:
blockchain-influxdb-local.influx.svc.cluster.local.numericable.fr   canonical name = nc-ass-vip.sdv.fr.
Name:   nc-ass-vip.sdv.fr
Address: 212.95.74.75

pod "dnsutils" deleted
pod default/dnsutils terminated (Error)
➜  ~ kubectl run dnsutils -it --rm=true --restart=Never --image=tutum/dnsutils -- nslookup -debug blockchain-influxdb-local.influx.svc.cluster.local.
Server:     10.96.0.10
Address:    10.96.0.10#53

------------
    QUESTIONS:
    blockchain-influxdb-local.influx.svc.cluster.local, type = A, class = IN
    ANSWERS:
    -&gt;  blockchain-influxdb-local.influx.svc.cluster.local
    internet address = 10.96.72.6
    ttl = 30
    AUTHORITY RECORDS:
    ADDITIONAL RECORDS:
------------
Name:   blockchain-influxdb-local.influx.svc.cluster.local
Address: 10.96.72.6

pod "dnsutils" deleted
</code></pre>
