<p>I think the title is pretty much self explanatory. I have done many experiments and the sad truth, is that <code>coredns</code> does add a <code>20 ms</code> overhead to all the requests inside the cluster. At first we thought maybe by adding more replications, and balancing the resolving requests between more instances, we could improve the response time, but it did not help at all. (we scaled up from 2 pods to 4 pods)</p>

<p>There was some enhancements on the <em>fluctuations</em> of resolving time, after scaling up to 4 instances. But it wasn't what we were expecting, and the <code>20 ms</code> overhead was still there.</p>

<p>We have some web-services that their actual response time is <code>&lt; 30 ms</code> and using <code>coredns</code> we are doubling up the response time, and it is not cool!</p>

<p>After coming to a conclusion about this matter, we did an experiment to double-check that this is not an OS level overhead. And the results were not different from what we were expecting.</p>

<p><a href="https://i.stack.imgur.com/oGChX.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/oGChX.png" alt=""></a>
<a href="https://i.stack.imgur.com/zadlE.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/zadlE.png" alt=""></a></p>

<p>We thought maybe we can implement/deploy a solution based on putting list of needed <code>hostname</code> mappings for each pod, inside <code>/etc/hosts</code> of that pod. So my final questions are as follows:</p>

<ul>
<li>Has anyone else experienced something similar with <code>coredns</code>?</li>
<li>Can you please suggest alternative solutions to <code>coredns</code> that work in k8s environment?</li>
</ul>

<p>Any thoughts or insights are appreciated. Thanks in advance.</p>

<p>There are several things to look at when running coreDNS in your kubernetes cluster </p>

<ul>
<li>Memory </li>
<li>AutoPath </li>
<li>Number of Replicas</li>
<li>Autoscaler</li>
<li>Other Plugins</li>
<li>Prometheus metrics </li>
<li>Separate Server blocks</li>
</ul>

<h3>Memory</h3>

<p>CoreDNS recommended amount of memory for replicas is </p>

<pre><code>MB required (default settings) = (Pods + Services) / 1000 + 54
</code></pre>

<p><a href="https://i.stack.imgur.com/KU8Jy.png" rel="noreferrer"><img src="https://i.stack.imgur.com/KU8Jy.png" alt="enter image description here"></a></p>

<h3>Autopath</h3>

<p><a href="https://github.com/coredns/coredns/tree/master/plugin/autopath" rel="noreferrer">Autopath</a> is a feature in Coredns that helps increase the response time for external queries </p>

<p>Normally a DNS query goes through </p>

<ol>
<li>..svc.cluster.local</li>
<li>.svc.cluster.local </li>
<li>cluster.local </li>
<li>Then the configured forward, usually host search path (/etc/resolv.conf</li>
</ol>

<pre><code>Trying "example.com.default.svc.cluster.local"
Trying "example.com.svc.cluster.local"
Trying "example.com.cluster.local"
Trying "example.com"
Trying "example.com"
;; -&gt;&gt;HEADER&lt;&lt;- opcode: QUERY, status: NOERROR, id: 55265
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;example.com.           IN  A

;; ANSWER SECTION:
example.com.        30  IN  A   93.184.216.34
</code></pre>

<p>This requires more memory so the calculation now becomes </p>

<pre><code>MB required (w/ autopath) = (Number of Pods + Services) / 250 + 56
</code></pre>

<h3>Number of replicas</h3>

<p>Defaults to 2 but enabling the Autoscaler should help with load issues. </p>

<h3>Autoscaler</h3>

<pre><code>apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: coredns
  namespace: default
spec:
  maxReplicas: 20
  minReplicas: 2
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: coredns
  targetCPUUtilizationPercentage: 50
</code></pre>

<h3>Node local cache</h3>

<p>Beta in Kubernetes 1.15 </p>

<blockquote>
  <p>NodeLocal DNSCache improves Cluster DNS performance by running a dns caching agent on cluster nodes as a DaemonSet. In today’s architecture, Pods in ClusterFirst DNS mode reach out to a kube-dns serviceIP for DNS queries. This is translated to a kube-dns/CoreDNS endpoint via iptables rules added by kube-proxy. With this new architecture, Pods will reach out to the dns caching agent running on the same node, thereby avoiding iptables DNAT rules and connection tracking. The local caching agent will query kube-dns service for cache misses of cluster hostnames(cluster.local suffix by default).</p>
</blockquote>

<p><a href="https://kubernetes.io/docs/tasks/administer-cluster/nodelocaldns/" rel="noreferrer">https://kubernetes.io/docs/tasks/administer-cluster/nodelocaldns/</a></p>

<h3>Other Plugins</h3>

<p>These will also help see what is going on inside CoreDNS</p>

<p><a href="https://coredns.io/plugins/errors/" rel="noreferrer">Error</a> - Any errors encountered during the query processing will be printed to standard output.</p>

<p><a href="https://coredns.io/plugins/trace/" rel="noreferrer">Trace</a> - enable OpenTracing of how a request flows through CoreDNS</p>

<p><a href="https://coredns.io/plugins/log/" rel="noreferrer">Log</a> - query logging</p>

<p><a href="https://coredns.io/plugins/health/" rel="noreferrer">health</a> - CoreDNS is up and running this returns a 200 OK HTTP status code</p>

<p><a href="https://coredns.io/plugins/ready/" rel="noreferrer">ready</a> - By enabling ready an HTTP endpoint on port 8181 will return 200 OK when all plugins that are able to signal readiness have done so.</p>

<p>Ready and Health should be used in the deployment </p>

<pre><code>        livenessProbe:
          httpGet:
            path: /health
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
        readinessProbe:
          httpGet:
            path: /ready
            port: 8181
            scheme: HTTP
</code></pre>

<h3>Prometheus Metrics</h3>

<p><a href="https://github.com/coredns/coredns/tree/master/plugin/metrics" rel="noreferrer">Prometheus Plugin</a></p>

<p><code>coredns_health_request_duration_seconds{} - duration to process a HTTP query to the local /health endpoint. As this a local operation, it should be fast. A (large) increase in this duration indicates the CoreDNS process is having trouble keeping up with its query load.</code></p>

<p><a href="https://github.com/coredns/deployment/blob/master/kubernetes/Scaling_CoreDNS.md" rel="noreferrer">https://github.com/coredns/deployment/blob/master/kubernetes/Scaling_CoreDNS.md</a></p>

<h3>Separate Server blocks</h3>

<p>One last bit of advice is to separate the Cluster DNS server block to external block </p>

<pre><code>    CLUSTER_DOMAIN REVERSE_CIDRS {
        errors
        health
        kubernetes
        ready
        prometheus :9153
        loop
        reload
        loadbalance
    }

    . {
        errors
        autopath @kubernetes
        forward . UPSTREAMNAMESERVER
        cache
        loop
    }
</code></pre>

<p>More information about the k8 plugin and other options here 
<a href="https://github.com/coredns/coredns/blob/master/plugin/kubernetes/README.md" rel="noreferrer">https://github.com/coredns/coredns/blob/master/plugin/kubernetes/README.md</a></p>
