<p>I am trying kubernetes and seem to have hit bit of a hurdle. The problem is that from within my pod I can't curl local hostnames such as <strong>wrkr1</strong> or <strong>wrkr2</strong> (machine hostnames on my network) but can successfully resolve hostnames such as google.com or stackoverflow.com.</p>
<p>My cluster is a basic setup with one master and 2 worker nodes.</p>
<p><strong>What works from within the pod:</strong></p>
<ul>
<li><p>curl to <strong>google.com</strong> from pod -- works</p>
</li>
<li><p>curl to another service(kubernetes) from pod -- works</p>
</li>
<li><p>curl to another machine on same LAN via its IP address such as 192.168.x.x -- works</p>
</li>
<li><p>curl to another machine on same LAN via its hostname such as wrkr1 -- does not work</p>
</li>
</ul>
<p><strong>What works from the node hosting pod:</strong></p>
<ul>
<li>curl to google.com --works</li>
<li>curl to another machine on same LAN via
its IP address such as 192.168.x.x -- works</li>
<li>curl to another machine
on same LAN via its hostname such as wrkr1 -- works.</li>
</ul>
<blockquote>
<p>Note: the pod cidr is completely different from the IP range used in
LAN</p>
</blockquote>
<p>the node contains a hosts file with entry corresponding to wrkr1's IP address (although I've checked node is able to resolve hostname without it also but I read somewhere that a pod inherits its nodes DNS resolution so I've kept the entry)</p>
<p>Kubernetes Version: <strong>1.19.14</strong></p>
<p>Ubuntu Version: <strong>18.04 LTS</strong></p>
<p>Need help as to whether this is normal behavior and what can be done if I want pod to be able to resolve hostnames on local LAN as well?</p>

<h2>What happens</h2>
<blockquote>
<p>Need help as to whether this is normal behavior</p>
</blockquote>
<p>This is normal behaviour, because there's no DNS server in your network where virtual machines are hosted and kubernetes has its own DNS server inside the cluster, it simply doesn't know about what happens on your host, especially in <code>/etc/hosts</code> because pods simply don't have access to this file.</p>
<blockquote>
<p>I read somewhere that a pod inherits its nodes DNS resolution so I've
kept the entry</p>
</blockquote>
<p>This is a point where tricky thing happens. <a href="https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-s-dns-policy" rel="nofollow noreferrer">There are four available</a> <code>DNS policies</code> which are applied per pod. We will take a look at two of them which are usually used:</p>
<ul>
<li>&quot;<strong>Default</strong>&quot;: The Pod inherits the name resolution configuration from the node that the pods run on. See related discussion for more details.</li>
<li>&quot;<strong>ClusterFirst</strong>&quot;: Any DNS query that does not match the configured cluster domain suffix, such as &quot;www.kubernetes.io&quot;, is forwarded to the upstream nameserver inherited from the node. Cluster administrators may have extra stub-domain and upstream DNS servers configured</li>
</ul>
<p>The trickiest ever part is this (from the same link above):</p>
<blockquote>
<p>Note: &quot;Default&quot; is not the default DNS policy. If dnsPolicy is not
explicitly specified, then &quot;ClusterFirst&quot; is used.</p>
</blockquote>
<p>That means that all pods that do not have <code>DNS policy</code> set will be run with <code>ClusterFirst</code> and they won't be able to see <code>/etc/resolv.conf</code> on the host. I tried changing this to <code>Default</code> and indeed, it can resolve everything host can, however internal resolving stops working, so it's not an option.</p>
<p>For example <code>coredns</code> deployment is run with <code>Default</code> dnsPolicy which allows <code>coredns</code> to resolve hosts.</p>
<h2>How this can be resolved</h2>
<p><strong>1. Add <code>local</code> domain to <code>coreDNS</code></strong></p>
<p>This will require to add <code>A</code> records per host. Here's a part from edited coredns configmap:</p>
<p>This should be within <code>.:53 {</code> block</p>
<pre><code>file /etc/coredns/local.record local
</code></pre>
<p>This part is right after block above ends (SOA information was taken from the example, it doesn't make any difference here):</p>
<pre><code>local.record: |
  local.            IN      SOA     sns.dns.icann.org. noc.dns.icann.org. 2015082541 7200 3600 1209600 3600
  wrkr1.            IN      A      172.10.10.10
  wrkr2.            IN      A      172.11.11.11
</code></pre>
<p>Then <code>coreDNS</code> deployment should be added to include this file:</p>
<pre><code>$ kubectl edit deploy coredns -n kube-system
      volumes:
      - configMap:
          defaultMode: 420
          items:
          - key: Corefile
            path: Corefile
          - key: local.record # 1st line to add
            path: local.record # 2nd line to add
          name: coredns
</code></pre>
<p>And restart <code>coreDNS</code> deployment:</p>
<pre><code>$ kubectl rollout restart deploy coredns -n kube-system
</code></pre>
<p>Just in case check if <code>coredns</code> pods are <code>running and ready</code>:</p>
<pre><code>$ kubectl get pods -A | grep coredns
kube-system   coredns-6ddbbfd76-mk2wv              1/1     Running            0                4h46m
kube-system   coredns-6ddbbfd76-ngrmq              1/1     Running            0                4h46m
</code></pre>
<p>If everything's done correctly, now newly created pods will be able to resolve hosts by their names. <a href="https://coredns.io/2017/05/08/custom-dns-entries-for-kubernetes/" rel="nofollow noreferrer">Please find an example in coredns documentation</a></p>
<p><strong>2. Set up DNS server in the network</strong></p>
<p>While <code>avahi</code> looks similar to DNS server, it does not act like a DNS server. It's not possible to setup <code>requests forwarding</code> from <code>coredns</code> to <code>avahi</code>, while it's possible to proper DNS server in the network and this way have everything will be resolved.</p>
<p><strong>3. Deploy <code>avahi</code> to kubernetes cluster</strong></p>
<p>There's a ready image with <code>avahi</code> <a href="https://hub.docker.com/r/flungo/avahi" rel="nofollow noreferrer">here</a>. If it's deployed into the cluster with <code>dnsPolicy</code> set to <code>ClusterFirstWithHostNet</code> and most importantly <code>hostNetwork: true</code> it will be able to use host adapter to discover all available hosts within the network.</p>
<h2>Useful links:</h2>
<ul>
<li><a href="https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-s-dns-policy" rel="nofollow noreferrer">Pods DNS policy</a></li>
<li><a href="https://coredns.io/2017/05/08/custom-dns-entries-for-kubernetes/" rel="nofollow noreferrer">Custom DNS entries for kubernetes</a></li>
</ul>
