<p>So this has been working forever.  I have a few simple services running in GKE and they refer to each other via the standard service.namespace DNS names.</p>

<p>Today all DNS name resolution stopped working.  I haven't changed anything, although this may have been triggered by a master upgrade.</p>

<pre><code>/ambassador # nslookup ambassador-monitor.default
nslookup: can't resolve '(null)': Name does not resolve

nslookup: can't resolve 'ambassador-monitor.default': Try again


/ambassador # cat /etc/resolv.conf  
search default.svc.cluster.local svc.cluster.local cluster.local c.snowcloud-01.internal google.internal 
nameserver 10.207.0.10 
options ndots:5
</code></pre>

<p>Master version  1.14.7-gke.14</p>

<p>I can talk cross-service using their IP addresses, it's just DNS that's not working.</p>

<p>Not really sure what to do about this...</p>

<p>The easiest way to verify if there is a problem with your Kube DNS is to look at the logs StackDriver [https://cloud.google.com/logging/docs/view/overview].</p>

<p>You should be able to find DNS resolution failures in the logs for the pods, with a filter such as the following:</p>

<p>resource.type="container" </p>

<p>("UnknownHost" OR "lookup fail" OR "gaierror") </p>

<p>&nbsp;</p>

<p>Be sure to check logs for each container. Because the exact names and numbers of containers can change with the GKE version, you can find them like so:</p>

<p> kubectl get pod -n kube-system -l k8s-app=kube-dns -o \ </p>

<p>jsonpath='{range .items[*].spec.containers[*]}{.name}{"\n"}{end}' | sort -u kubectl get pods -n kube-system -l k8s-app=kube-dns </p>

<p>&nbsp;</p>

<p>Has the pod been restarted frequently? Look for OOMs in the node console. The nodes for each pod can be found like so:</p>

<p>kubectl get pod -n kube-system -l k8s-app=kube-dns -o \ </p>

<p>jsonpath='{range .items[*]}{.spec.nodeName} pod={.metadata.name}{"\n"}{end}' </p>

<p>&nbsp;</p>

<p>The&nbsp;<code>kube-dns</code>&nbsp;pod contains four containers:</p>

<ul>
<li><code>kube-dns</code>&nbsp;process watches the Kubernetes master for changes in Services and Endpoints, and maintains in-memory lookup structures to serve DNS requests,</li>
<li><code>dnsmasq</code>&nbsp;adds DNS caching to improve performance,</li>
<li><code>sidecar</code>&nbsp;provides a single health check endpoint while performing dual health checks (for&nbsp;<code>dnsmasq</code>&nbsp;and&nbsp;<code>kubedns</code>). It also collects dnsmasq metrics and exposes them in the Prometheus format,</li>
<li><code>prometheus-to-sd</code>&nbsp;scraping the metrics exposed by&nbsp;<code>sidecar</code>&nbsp;and sending them to Stackdriver.</li>
</ul>

<p>By default, the&nbsp;<code>dnsmasq</code>&nbsp;container accepts 150 concurrent requests. Requests beyond this are simply dropped and result in failed DNS resolution, including resolution for&nbsp;<code>metadata</code>. To check for this, view the logs with the following filter:</p>

<p>resource.type="container"<br />resource.labels.cluster_name="&lt;cluster-name&gt;"<br />resource.labels.namespace_id="kube-system"<br />logName="projects/&lt;project-id&gt;/logs/dnsmasq"<br />"Maximum number of concurrent DNS queries reached"</p>

<p>&nbsp;</p>

<p>If legacy stackdriver logging of cluster is disabled, use the following filter:</p>

<p>resource.type="k8s_container"<br />resource.labels.cluster_name="&lt;cluster-name&gt;"<br />resource.labels.namespace_name="kube-system"<br />resource.labels.container_name="dnsmasq"<br />"Maximum number of concurrent DNS queries reached"</p>

<p>&nbsp;</p>

<p>If Stackdriver logging is disabled, execute the following:</p>

<p>kubectl logs --tail=1000 --namespace=kube-system -l k8s-app=kube-dns -c dnsmasq | grep 'Maximum number of concurrent DNS queries reached'</p>

<p>&nbsp;</p>

<p>Additionally, you can try to use the command [dig ambassador-monitor.default @10.207.0.10] from each nodes to verify if this is only impacting one node. If it is, you can simple re-create the impacted node.</p>
