<p>I'm running my Backend on <strong>Kubernetes</strong> on around <strong>250</strong> pods under <strong>15</strong> deployments, backend in written in <strong>NODEJS</strong>.</p>
<p>Sometimes after X number of hours (5&lt;X&lt;30) I'm getting <code>ENOTFOUND</code> in one of the PODS, as follows:</p>
<pre><code>{
  &quot;name&quot;: &quot;main&quot;,
  &quot;hostname&quot;: &quot;entrypoint-sdk-54c8788caa-aa3cj&quot;,
  &quot;pid&quot;: 19,
  &quot;level&quot;: 50,
  &quot;error&quot;: {
    &quot;errno&quot;: -3008,
    &quot;code&quot;: &quot;ENOTFOUND&quot;,
    &quot;syscall&quot;: &quot;getaddrinfo&quot;,
    &quot;hostname&quot;: &quot;employees-service&quot;
  },
  &quot;msg&quot;: &quot;Failed calling getEmployee&quot;,
  &quot;time&quot;: &quot;2022-01-28T13:44:36.549Z&quot;,
  &quot;v&quot;: 0
}
</code></pre>
<p>I'm running a stress test on the Backend of YY number of users per second, but I'm keeping this stress level steady and not changing it, and then it happens out of nowhere with no specific reason.</p>
<p>Kubernetes is <strong>K3S</strong> Server Version: <strong>v1.21.5+k3s2</strong></p>
<p>Any idea what might cause this weird <code>ENOTFOUND</code>?</p>

JAN
<p>Already saw your <a href="https://github.com/kubernetes/kubernetes/issues/107866" rel="nofollow noreferrer">same question on github</a> and reference to <a href="https://github.com/external-secrets/kubernetes-external-secrets/issues/860" rel="nofollow noreferrer">getaddrinfo ENOTFOUND with newest versions</a>.</p>
<p>As per comments this issue does not appear in k3s 1.21, that is 1 version below yours. I know it almost impossible, but any chance to try similar setup on this ver?</p>
<p>And it seems error comes from <a href="https://github.com/nodejs/node/blob/1125c8a814030f16fc995d897181ac9920f9d3db/lib/dns.js#L17-L24" rel="nofollow noreferrer">node/lib/dns.js</a>.</p>
<pre><code>function errnoException(err, syscall, hostname) {
  // FIXME(bnoordhuis) Remove this backwards compatibility nonsense and pass
  // the true error to the user. ENOTFOUND is not even a proper POSIX error!
  if (err === uv.UV_EAI_MEMORY ||
      err === uv.UV_EAI_NODATA ||
      err === uv.UV_EAI_NONAME) {
    err = 'ENOTFOUND';
  }
</code></pre>
<p>What I wanted to suggest you is to check <a href="https://tech.findmypast.com/k8s-dns-lookup/" rel="nofollow noreferrer">Solving DNS lookup failures in Kubernetes</a>. Article describes long hard way of catching the same error you have that also  bothered from time to time.</p>
<p>As a solution aftet investigating all the metrics, logs, etc - was installing   K8s cluster add-on called <a href="https://kubernetes.io/docs/tasks/administer-cluster/nodelocaldns/" rel="nofollow noreferrer">Node Local DNS cache</a>, that</p>
<blockquote>
<p>improves Cluster DNS performance by running a dns
caching agent on cluster nodes as a DaemonSet. In today's
architecture, Pods in ClusterFirst DNS mode reach out to a kube-dns
serviceIP for DNS queries. This is translated to a kube-dns/CoreDNS
endpoint via iptables rules added by kube-proxy. With this new
architecture, Pods will reach out to the dns caching agent running on
the same node, thereby avoiding iptables DNAT rules and connection
tracking. The local caching agent will query kube-dns service for
cache misses of cluster hostnames(cluster.local suffix by default).</p>
<p>Motivation</p>
<ul>
<li>With the current DNS architecture, it is possible that Pods with the    highest DNS QPS have to reach out to a different node, if there
is no    local kube-dns/CoreDNS instance. Having a local cache will
help    improve the latency in such scenarios.</li>
<li>Skipping iptables DNAT and connection tracking will help reduce    conntrack races and avoid UDP DNS entries filling up conntrack table.</li>
<li>Connections from local caching agent to kube-dns service can be    upgraded to TCP. TCP conntrack entries will be removed on connection<br />
close in contrast with UDP entries that have to timeout (default<br />
nf_conntrack_udp_timeout is 30 seconds)</li>
<li>Upgrading DNS queries from UDP to TCP would reduce tail latency    attributed to dropped UDP packets and DNS timeouts usually up to 30s<br />
(3 retries + 10s timeout). Since the nodelocal cache listens for UDP<br />
DNS queries, applications don't need to be changed.</li>
<li>Metrics &amp; visibility into dns requests at a node level.</li>
<li>Negative caching can be re-enabled, thereby reducing number of    queries to kube-dns service.</li>
</ul>
</blockquote>
