<p>We have an Apache Airflow deployed over a K8s cluster in AWS. Airflow is running on containers but the <code>EC2</code> instances themselves are reserved instances.</p>
<p>We are experiencing an issue where we see that Ariflow is making many DNS queries related to it's DB. When at rest (i.e. no DAGs are running) it's about <strong>10</strong> per second. When running several DAGs it can go up to <strong>50</strong> per second. This results in <code>Route53</code> blocking us since we are hitting the packet limit for DNS queries (1024 packets per second).</p>
<p>Our DB is a <code>Postgres</code> RDS, and when switching it to a <code>MySQL</code> the issue remains.</p>
<p>The way we understand it, the DNS query starts at K8s <code>coredns</code> service, which tries several permutations of the FQDN and sends the requests to <code>Route53</code> if it can't resolve it on it's own.</p>
<p>Any ideas, thoughts, or hints to explain Airflow's behavior or how to reduce the number of queries is most welcome.</p>
<p>Best,</p>

Guy Grin
<p>After some digging we found we had several issues happening at the same time.</p>
<p>The first being that Airflow's scheduler was running about 2 times per second. Each time it created DB queries which resulted in several DNS queries. Changing that scheduling alleviated some of the issue.</p>
<p>Another issue we had is described <a href="https://pracucci.com/kubernetes-dns-resolution-ndots-options-and-why-it-may-affect-application-performances.html" rel="nofollow noreferrer">here</a>. It looks like <code>coredns</code> is configured to try some alternatives of the given domain if it has less than <code>x</code> number of <code>.</code> in the FQDN. There are 2 suggested fixes in that article. We followed them through and the number of DNS queries dropped.</p>
