<p>I cannot resolve service from kubernetes.</p>

<pre><code>kubectl get pods -l k8s-app=kube-dns --namespace kube-system
NAME                       READY   STATUS    RESTARTS   AGE   IP
coredns-86c58d9df4-gn62b   1/1     Running   0          18d   10.244.0.58
coredns-86c58d9df4-svmk5   1/1     Running   0          18d   10.244.0.59
</code></pre>

<p>containers do not resolve any domains, including kubernetes.default</p>

<pre><code>kubectl exec -ti busybox -- sh
/ # nslookup kubernetes.
defaultServer:    10.96.0.10
Address 1: 10.96.0.10

nslookup: can't resolve 'kubernetes.default'
command terminated with exit code 1
</code></pre>

<p>Logs from dns pods do not show any queries (note coredns is configured to log queries)</p>

<pre><code>kubectl logs --namespace=kube-system coredns-86c58d9df4-gn62b
.:53
2019-01-18T21:44:34.271Z [INFO] CoreDNS-1.2.6
2019-01-18T21:44:34.271Z [INFO] linux/amd64, go1.11.2, 756749c
CoreDNS-1.2.6
linux/amd64, go1.11.2, 756749c
 [INFO] plugin/reload: Running configuration MD5 = f65c4821c8a9b7b5eb30fa4fbc167769
[INFO] Reloading
 [INFO] plugin/reload: Running configuration MD5 = 2394cf331ea25e9aacc36ddf69fafcdb
[INFO] Reloading complete
2019-02-04T22:23:21.266Z [INFO] 127.0.0.1:39695 - 58939 "HINFO IN 4718439545634584094.2038959545847864189. udp 57 false 512" NXDOMAIN qr,rd,ra 133 0.021492508s
</code></pre>

<p>The kube-node coredns is hosted on is running ubuntu xenial. </p>

<p><a href="https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/#known-issues" rel="nofollow noreferrer">I noticed there is a known issue on ubuntu hosts</a></p>

<p>I applied custom kubelet conf, setting <code>--resolv-conf=/run/systemd/resolve/resolv.conf</code> </p>

<pre><code>❯ systemctl status kubelet.service
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since Wed 2019-02-06 01:05:42 GMT; 5min ago
     Docs: https://kubernetes.io/docs/home/
 Main PID: 27867 (kubelet)
    Tasks: 30 (limit: 4915)
   CGroup: /system.slice/kubelet.service
           └─27867 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-drive --resolv-conf=/run/systemd/resolve/resolv.conf
</code></pre>

<p>However I still cannot resolve any services. </p>

<p>I delete the pods, and their controller rescheduled them.</p>

<p>Now dns queries and service discovery is working.</p>

<p>Am not sure if the coredns service is now reachable because I update <code>kubelet --resolve-conf</code> or if the service just needed to restart.</p>
