<p>We have a service that sends tons of events in bulks. It basically opens multiple http POST connections.</p>
<p>Since we moved the service to kubernetes, we're <code>getaddrinfo: Temporary failure in name resolution</code> errors from time to time. (most calls work but some fail and it's weird.</p>
<p>Can anyone explain why and how to fix?</p>

<p>Check the tinder post, they had a similar problem:</p>

<p><a href="https://medium.com/tinder-engineering/tinders-move-to-kubernetes-cda2a6372f44" rel="nofollow noreferrer">https://medium.com/tinder-engineering/tinders-move-to-kubernetes-cda2a6372f44</a></p>

<p>and the source for their dns info:</p>

<p><a href="https://www.weave.works/blog/racy-conntrack-and-dns-lookup-timeouts" rel="nofollow noreferrer">https://www.weave.works/blog/racy-conntrack-and-dns-lookup-timeouts</a></p>

<p>TLDR: check your arp tables cache gc_* host parameters, try to disable AAAA query in the containers /etc/gai.conf, move the DNS to a daemonset and inject the host IP as dns servers to the pods</p>

<p>Also, to help this and speed up dns resolve, add a ending dot to all domains (ie: database.example.com.), so coredns will try that query directly (one query, 2 with ipv6) instead of trying all the kubernetes search domain list ( about 5, 10 with ipv6). Only leave out that dot where you are querying for kubernetes resources or in apps that do not like it (if they exist, it's a bug, ALL DNS always end with a dot, while most of the time we can leave it out, with it is the correct way, must not fail)</p>
