<p>Having some issue with my kubernetes cluster and DNS.</p>

<p>We recently updated to RHEL 7.5 and one of the machines was the Master. Once it back online most everything worked but I just noticed that external connections from the cluster do not resolve. Internal communication works great.</p>

<p>Here's the busybox nslookup results:</p>

<pre><code>kubectl exec -ti busybox -- nslookup kubernetes.default
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

nslookup: can't resolve 'kubernetes.default'
</code></pre>

<p>kubedns status</p>

<pre><code>Name:              kube-dns
Namespace:         kube-system
Labels:            k8s-app=kube-dns
                   kubernetes.io/cluster-service=true
                   kubernetes.io/name=KubeDNS
Annotations:       &lt;none&gt;
Selector:          k8s-app=kube-dns
Type:              ClusterIP
IP:                10.96.0.10
Port:              dns  53/UDP
TargetPort:        53/UDP
Endpoints:         10.32.0.18:53
Port:              dns-tcp  53/TCP
TargetPort:        53/TCP
Endpoints:         10.32.0.18:53
Session Affinity:  None
Events:            &lt;none&gt;
</code></pre>

<p>All pods say they are up</p>

<pre><code>NAME                        READY     STATUS    RESTARTS   AGE
kube-dns-86f4d74b45-9m292   3/3       Running   26         44d
</code></pre>

<p>Lastest logs from the kubedns pods</p>

<pre><code>I0523 15:59:14.291623       1 logs.go:41] skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053 [rcache 0]
I0523 15:59:14.291638       1 logs.go:41] skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053 [rcache 0]
I0523 15:59:14.791440       1 dns.go:170] Initialized services and endpoints from apiserver
I0523 15:59:14.791560       1 server.go:135] Setting up Healthz Handler (/readiness)
I0523 15:59:14.791579       1 server.go:140] Setting up cache handler (/cache)
I0523 15:59:14.791588       1 server.go:126] Status HTTP port 8081
I0523 16:48:05.175159       1 dns.go:555] Could not find endpoints for service "kube-prometheus-exporter-kube-controller-manager" in namespace "kube-system". DNS records will be created once endpoints show up.
I0523 16:48:05.236418       1 dns.go:555] Could not find endpoints for service "kube-prometheus-exporter-kube-dns" in namespace "kube-system". DNS records will be created once endpoints show up.
I0523 16:48:05.252139       1 dns.go:555] Could not find endpoints for service "kube-prometheus-exporter-kube-etcd" in namespace "kube-system". DNS records will be created once endpoints show up.
I0523 16:48:05.264292       1 dns.go:555] Could not find endpoints for service "kube-prometheus-exporter-kube-scheduler" in namespace "kube-system". DNS records will be created once endpoints show up.
I0524 13:04:38.008038       1 logs.go:41] skydns: failure to forward request "read udp 10.32.0.18:48431-&gt;10.101.24.100:53: i/o timeout"
I0524 13:04:41.973900       1 logs.go:41] skydns: failure to forward request "read udp 10.32.0.18:45738-&gt;10.101.24.100:53: i/o timeout"
I0524 17:26:54.056913       1 logs.go:41] skydns: failure to forward request "read udp 10.32.0.18:52242-&gt;10.101.24.100:53: i/o timeout"
I0524 18:28:16.762002       1 logs.go:41] skydns: failure to forward request "read udp 10.32.0.18:43051-&gt;10.101.24.100:53: i/o timeout"
I0524 18:28:20.762912       1 logs.go:41] skydns: failure to forward request "read udp 10.32.0.18:57624-&gt;10.101.24.100:53: i/o timeout"
</code></pre>

<p>I am assuming this has something to do with the reboot.</p>

<p>Kubernetes: version 1.10.3</p>

<p>DNS: kube-dns</p>

<p>Network: Weavenet</p>

<p>Host OS: RHEL 7.5</p>

<p>Any help would be great.</p>

<p>Thanks</p>

macintoshPrime
<p>Not sure of the details of your upgrade, but check to see if your kube-dns version changed when you upgraded your OS. There’s an issue with kube-dns version 1.14.9 resolving external name services. You’ll need to roll back kube-dns to a previous build, or upgrade to 1.14.10.</p>

<p>See this issue: <a href="https://github.com/kubernetes/kops/issues/4986" rel="nofollow noreferrer">https://github.com/kubernetes/kops/issues/4986</a></p>
