<p>I have a Azure Kubernetes Cluster with all pods and services in running state. The issue I have is when I do a curl from pod1 to the service url of pod2, it fails intermittently with a Unable to resolve host error. </p>

<p>To illustrate, I have 3 pods - pod1, pod2, pod3 
When I get into pod1 using </p>

<blockquote>
  <p>kubectl exec -it pod1</p>
</blockquote>

<p>and I run curl using service url of pod2 :</p>

<blockquote>
  <p>curl <a href="http://api-batchprocessing:3000/result" rel="nofollow noreferrer">http://api-batchprocessing:3000/result</a></p>
</blockquote>

<p>the command succeeds about every 6/10 times, the remaining 4/10 it fails with error "<code>curl: (6) Could not resolve host:api-batchprocessing</code>".  </p>

<p>When I tried calling another service running on pod3 using curl, I get the same issue.</p>

<p>I have tried below approaches without any success:
- delete coredns pods in kube-system 
- delete and recreate azure kubernetes cluster.
above seem to resolve it temporarily, but in few tries I get the same intermittent 'could not resolve host:' issue.</p>

<p>Any help/pointers on this issue  will be much appreciated.</p>

<p>Problem may lay in <a href="https://www.cloudflare.com/learning/dns/what-is-dns/" rel="nofollow noreferrer">DNS</a> configuration.
Looks like coredns uses the DNS server list in a differnt way that kube-dns did. 
If you have to resolve both public and private hostnames, always check only private DNS servers are on the list, or find the right configuration to route private DNS queries against your private premises.</p>

<p>Possible steps to find and get rid of problem:</p>

<ol>
<li>Turn coredns logs on.</li>
</ol>

<p>The only thing you need is this <a href="https://learn.getgrav.org/16/advanced/yaml" rel="nofollow noreferrer">YAML</a> file:</p>

<pre><code>apiVersion: v1
data:
  log.override: |
    log
kind: ConfigMap
metadata:
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
  name: coredns-custom
  namespace: kube-system
</code></pre>

<ol start="2">
<li><p>Use kubectl log or VSCode Kubernetes extension to open/review your coredns logs.</p></li>
<li><p>Attach to one of our environment pods and executed some DNS resolution actions, including nslookup and curl. 
Some of the executions where loop queries to put pressure on the DNS and networking components:</p></li>
<li><p>Review coredns the logs.</p></li>
</ol>

<p>You will see before, curl is trying to resolve DNS by trying both A and AAAA entries, for all the search domains defined in our pods. In other words, to resolve "api-batchprocessing", curl is making  DNS queries agains coredns. However, coredns is responding properly with an "NXDOMAIN" (no exist) or "NOERROR" (record found). So the problem is elsewhere.</p>

<ol start="5">
<li>Configure DNS servers at VNET level</li>
</ol>

<p>Possible explanation to this random DNS resolution error is that, under high load scenarios, coredns is using on all the DNS servers defined at VNET level.
Some queries were probably going to the on-prem servers. Others, to Google. Google doesn't know how to resolve your private hostnames.</p>

<ol start="6">
<li>Remove the Google DNS servers from the VNET, restar the cluster and run checks.</li>
</ol>

<p>Here you can find more information: <a href="https://www.sesispla.net/en/azure-aks-1-11-random-dns-resolution-error-after-cluster-upgrade/" rel="nofollow noreferrer">random-dns-error</a>.</p>

<p>I hope it helps.</p>
