<p>Lately, we've faced some DNS issues with micro-services based on Alpine image (node:12.18.1-alpine) on EKS when trying to resolve &quot;big&quot; DNS queries (When the answer is larger than 512M).</p>
<p>So I've tried running this script for testing the DNS resolution:</p>
<pre><code>var dns = require('dns');
var w3 = dns.lookup('hugedns.test.dziemba.net', function (err, addresses, family) {
  console.log(addresses);
});
</code></pre>
<p>with 2 different scenarios for each image</p>
<ol>
<li><strong>node:12.18.1-alpine</strong></li>
</ol>
<ul>
<li>Running the image on my laptop - Resolved successfully</li>
<li>Running the image on EKS 1.16 - Failed to resolve</li>
</ul>
<ol start="2">
<li><strong>node:12.18.1-slim</strong></li>
</ol>
<ul>
<li>Running the image on my laptop - Resolved successfully</li>
<li>Running the image on EKS 1.16 - Resolved successfully</li>
</ul>
<p>From what I saw, Alpine is using musl (which doesn't support DNS to use TCP?) libraries instead of glibc, since the DNS protocol is using UDP and tries falling back to TCP only when the query is larger than 512M. So my theory was that this is the root cause, but since it is working on my end and failing on EKS made me wonder where can the issue relay...</p>
<p>Any thoughts?</p>
<p>EKS v1.16
coredns:v1.6.6</p>
<p>BTW, this is my first post, let me know if any information is needed</p>

Shahaf Yehezkel
<p>Yes, the Alpine images are known to be problematic in Kubernetes cluster concerning DNS queries.</p>
<p>Even if it is not clear if the bug has been effectively fixed in any current version of Alpine, here are some related links:</p>
<ul>
<li><a href="https://github.com/kubernetes-sigs/kind/issues/442" rel="noreferrer">https://github.com/kubernetes-sigs/kind/issues/442</a></li>
<li><a href="https://gitlab.alpinelinux.org/alpine/aports/-/issues/9017" rel="noreferrer">https://gitlab.alpinelinux.org/alpine/aports/-/issues/9017</a></li>
<li><a href="https://github.com/gliderlabs/docker-alpine/issues/476" rel="noreferrer">https://github.com/gliderlabs/docker-alpine/issues/476</a></li>
</ul>
<p>I encountered this problem on my side in my Kubernetes clusters as of January 2021 with up-to-date Alpine 3.12 images, so I would assume it is not fixed.</p>
<p>The core problem seems to be that the <code>musl</code> library stop searching among possible domains specified in the <code>search</code> directive of <code>/etc/resolv.conf</code> for a given name if any response is unexpected (basically not something clearly indicating that the FQDN could not be found, or has been found).</p>
<p>This does not play well with the Kubernetes strategy about name resolution in pods.</p>
<p>Indeed, one can see that the typical <code>/etc/resolv.conf</code> of a pod in the <code>example</code> namespace is the following:</p>
<pre><code>nameserver 10.3.0.10
search example.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
</code></pre>
<p>The strategy is that the resolution of a name, for instance <code>my-service</code> or <code>www.google.com</code>, will be tested against each of the domains specified in the <code>search</code> directive: here for the examples, it would be the FQDN chains <code>my-service.example.svc.cluster.local</code>,<code>my-service.svc.cluster.local</code>,<code>my-service.cluster.local</code>,<code>my-service</code> and <code>www.google.com.example.svc.cluster.local</code>,<code>www.google.com.svc.cluster.local</code>,<code>www.google.com.cluster.local</code>,<code>www.google.com</code>. Here obviously it would be the 1st FQDN of the first chain (<code>my-service.example.svc.cluster.local</code>) and the last FQDN of the second chain (<code>www.google.com</code>) that would be resolved correctly.</p>
<p>One can see that this strategy is made to optimize resolution of internal name of the cluster, in a way that allows names like <code>my-service</code>, <code>my-service.my-namespace</code> or <code>my-service.my-namespace.svc</code> to be nicely resolved out-of-the-box.</p>
<p>The <code>ndots</code> parameter in the <code>options</code> directive defines the minimum number of dots in a name to consider that a name is actually a FQDN and so the search chain should be skiped in favor of a direct DNS resolution attempt. With <code>ndots:2</code>, <code>www.google.com</code> will be considered as a FQDN while <code>my-service.my-namespace</code> will go through the search chain.</p>
<p>Given that the <code>search</code> option over 3 possible domains, that any obvious URL will not be considered as a FQDN because of <code>ndots:5</code> and the break of the search loop in <code>musl</code> library in Alpine docker, all of this dramatically increases the probability of a host resolution failure in a Docker Alpine running in Kubernetes. If your host resolution is part of some kind of a loop running regularly, you will encounter a lot of failures that need to be handled.</p>
<p>What to do about this ?</p>
<ul>
<li>you can use a <code>dnsPolicy</code> to reduce <code>ndots</code> and consider shorter names as FQDN and skip the search loop (see <a href="https://pracucci.com/kubernetes-dns-resolution-ndots-options-and-why-it-may-affect-application-performances.html" rel="noreferrer">https://pracucci.com/kubernetes-dns-resolution-ndots-options-and-why-it-may-affect-application-performances.html</a>)</li>
<li>you can generate a entrypoint script for your image that would modify the <code>/etc/resolv.conf</code> accordingly to your needs, for instance <code>cat /etc/resolv.conf | sed -r &quot;s/^(search.*|options.*)/#\1/&quot; &gt; /tmp/resolv &amp;&amp; cat /tmp/resolv &gt; /etc/resolv.conf</code> will remove all the stuff about <code>search</code> and <code>options</code>, if you do not rely on any internal name of the cluster in your process</li>
<li>direct edition of <code>/etc/host</code> to hardcode some FQDN to their known IPs</li>
<li>move away from Alpine images, and go for Ubi or Debian/Ubuntu based images</li>
</ul>
<p>Personally I started with Alpine, like a lot of us in the early days of Docker industrialization because the other full OS images were insanely big. This is mostly not the case anymore with strongly tested slim images for Ubuntu or Debian, or even Kubernetes-centric initiatives like Ubi. That is why I usually choose the last alternative (move away from Alpine images).</p>
