<p>So I recently setup a single node kubernetes cluster on my home network. I have a dns server that runs on my router (DD-WRT, dnsmaq) that resolves a bunch of local domains for ease of use. <code>server1.lan</code>, for example resolves to <code>192.168.1.11</code>. </p>

<p>Server 1 was setup as my single node kubernetes cluster. Excited about the possibilities of local DNS, I spun up my first deployment using a docker container called netshoot which has a bunch of helpful network debugging tools bundled in. I execd into the container, and ran a ping and got the following...</p>

<pre><code>bash-5.0# ping server1.lan
ping: server1.lan: Try again
</code></pre>

<p>It failed, then I tried pinging google's DNS (8.8.8.8) and that worked fine.</p>

<p>I tried to resolve the kubernetes default domain, it worked fine</p>

<pre><code>bash-5.0# nslookup kubernetes.default
Server:     10.96.0.10
Address:    10.96.0.10#53

Name:   kubernetes.default.svc.cluster.local
Address: 10.96.0.1
</code></pre>

<p>The <code>/etc/resolve.conf</code> file looks fine from inside the pod</p>

<pre><code>bash-5.0# cat /etc/resolv.conf
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
</code></pre>

<p>I then got to tailing the coredns logs, and I started seeing some interesting output...</p>

<pre><code>2019-11-13T03:01:23.014Z [ERROR] plugin/errors: 2 server1.lan. AAAA: read udp 192.168.156.140:37521-&gt;192.168.1.1:53: i/o timeout
2019-11-13T03:01:24.515Z [ERROR] plugin/errors: 2 server1.lan. A: read udp 192.168.156.140:41964-&gt;192.168.1.1:53: i/o timeout
2019-11-13T03:01:24.515Z [ERROR] plugin/errors: 2 server1.lan. AAAA: read udp 192.168.156.140:33455-&gt;192.168.1.1:53: i/o timeout
2019-11-13T03:01:25.015Z [ERROR] plugin/errors: 2 server1.lan. AAAA: read udp 192.168.156.140:48864-&gt;192.168.1.1:53: i/o timeout
2019-11-13T03:01:25.015Z [ERROR] plugin/errors: 2 server1.lan. A: read udp 192.168.156.140:35328-&gt;192.168.1.1:53: i/o timeout
</code></pre>

<p>It seems like kubernetes is trying to communicate with <code>192.168.1.1</code> from inside the cluster network and failing. I guess CoreDNS uses whatever is in the <code>resolv.conf</code> on the host, so here is what that looks like.</p>

<pre><code>nameserver 192.168.1.1
</code></pre>

<p>I can resolve server1.lan from everywhere else on the network, except these pods. My router IP is 192.168.1.1, and that is what is responding to DNS queries.</p>

<p>Any help on this would be greatly appreciated, it seems like some kind of IP routing issue between the kubernetes network and my real home network, or that's my theory anyways. Thanks in advance.</p>

Kulix
<p>So it turns out the issue was that when I initiated the cluster, I specified a pod CIDR that conflicted with IPs on my home network. My kubeadm command was this</p>

<pre><code>sudo kubeadm init --pod-network-cidr=192.168.0.0/16 --apiserver-cert-extra-sans=server1.lan
</code></pre>

<p>Since my home network conflicted with that CIDR, and since my dns upstream was 192.168.1.1, it thought that was on the pod network and not on my home network and failed to route the DNS resolution packets appropriately. </p>

<p>The solution was to recreate my cluster using the following command, </p>

<pre><code>sudo kubeadm init --pod-network-cidr=10.200.0.0/16 --apiserver-cert-extra-sans=server1.lan
</code></pre>

<p>And when I applied my calico yaml file, I made sure to replace the default <code>192.168.0.0/16</code> CIDR with the new <code>10.200.0.0/16</code> CIDR.</p>

<p>Hope this helps someone. Thanks.</p>
