<p>I have a 4 node Kubernetes cluster, 1 x controller and 3 x workers. The following shows how they are configured with the versions.</p>

<p><code>
NAME             STATUS    ROLES     AGE       VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
k8s-ctrl-1       Ready     master    1h        v1.11.2   192.168.191.100       &lt;none&gt;        Ubuntu 18.04.1 LTS   4.15.0-1021-aws     docker://18.6.1
turtle-host-01   Ready     &lt;none&gt;    1h        v1.11.2   192.168.191.53   &lt;none&gt;        Ubuntu 18.04.1 LTS   4.15.0-29-generic   docker://18.6.1
turtle-host-02   Ready     &lt;none&gt;    1h        v1.11.2   192.168.191.2    &lt;none&gt;        Ubuntu 18.04.1 LTS   4.15.0-34-generic   docker://18.6.1
turtle-host-03   Ready     &lt;none&gt;    1h        v1.11.2   192.168.191.3    &lt;none&gt;        Ubuntu 18.04.1 LTS   4.15.0-33-generic   docker://18.6.1
</code></p>

<p>Each of the nodes has two network interfaces, for arguments sake <code>eth0</code> and <code>eth1</code>. <code>eth1</code> is the network that I want to the cluster to work on. I setup the controller using <code>kubeadm init</code> and passed <code>--api-advertise-address 192.168.191.100</code>. The worker nodes where then joined using this address.</p>

<p>Finally on each node I modified the kubelet service to have <code>--node-ip</code> set so that the layout looks as above.</p>

<p>The cluster appears to be working correctly and I can create pods, deployments etc. However the issue I have is that none of the pods are able to use the <code>kube-dns</code> service for DNS resolution.</p>

<p>This is not a problem with resolution, rather that the machines cannot connect to the DNS service to perform the resolution. For example if I run a <code>busybox</code> container and access it to perform <code>nslookup</code> i get the following:</p>

<p><code>
/ # nslookup www.google.co.uk
nslookup: read: Connection refused
nslookup: write to '10.96.0.10': Connection refused
</code></p>

<p>I have a feeling that this is down to not using the default network and because of that I suspect some Iptables rules are not correct, that being said these are just guesses.</p>

<p>I have tried both the Flannel overlay and now Weave net. The pod CIDR range is <code>10.32.0.0/16</code> and the service CIDR is as default.</p>

<p>I have noticed that with Kubernetes 1.11 there are now pods called <code>coredns</code> rather than one <code>kube-dns</code>.</p>

<p>I hope that this is a good place to ask this question. I am sure I am missing something small but vital so if anyone has any ideas that would be most welcome.</p>

<p><strong>Update #1:</strong></p>

<p>I should have said that the nodes are not all in the same place. I have a VPN running between them all and this is the network I want things to communicate over. It is an idea I had to try and have distributed nodes.</p>

<p><strong>Update #2:</strong></p>

<p>I saw another answer on SO (<a href="https://stackoverflow.com/questions/30992961/dns-in-kubernetes-not-working?answertab=active#tab-top">DNS in Kubernetes not working</a>) that suggested <code>kubelet</code> needed to have <code>--cluster-dns</code> and <code>--cluster-domain</code> set. This is indeed the case on my DEV K8s cluster that I have running at home (on one network).</p>

<p>However it is not the case on this cluster and I suspect this is down to a later version. I did add the two settings to all nodes in the cluster, but it did not make things work.</p>

<p><strong>Update #3</strong></p>

<p>The topology of the cluster is as follows.</p>

<ul>
<li>1 x Controller is in AWS</li>
<li>1 x Worker is in Azure</li>
<li>2 x Worker are physical machines in a colo Data Centre</li>
</ul>

<p>All machines are connected to each other using ZeroTier VPN on the 192.168.191.0/24 network.</p>

<p>I have <em>not</em> configured any special routing. I agree that this is probably where the issue is, but I am not 100% sure what this routing should be.</p>

<p>WRT to <code>kube-dns</code> and <code>nginx</code>, I have not tainted my controller so <code>nginx</code> is not on the master, not is <code>busybox</code>. <code>nginx</code> and <code>busybox</code> are on workers 1 and 2 respectively.</p>

<p>I have used <code>netcat</code> to test connection to <code>kube-dns</code> and I get the following:</p>

<p><code>
/ # nc -vv 10.96.0.10 53
nc: 10.96.0.10 (10.96.0.10:53): Connection refused
sent 0, rcvd 0
/ # nc -uvv 10.96.0.10 53
10.96.0.10 (10.96.0.10:53) open
</code></p>

<p>The UDP connection does not complete.</p>

<p>I modified my setup so that I could run containers on the controller, so <code>kube-dns</code>, <code>nginx</code> and <code>busybox</code> are all on the controller, and I am able to connect and resolve DNS queries against 10.96.0.10.</p>

<p>So all this does point to routing or IPTables IMHO, I just need to work out what that should be.</p>

<p><strong>Update #4</strong></p>

<p>In response to comments I can confirm the following ping test results.</p>

<pre><code>Master -&gt; Azure Worker (Internet)  : SUCCESS : Traceroute SUCCESS
Master -&gt; Azure Worker (VPN)       : SUCCESS : Traceroute SUCCESS
Azure Worker -&gt; Master (Internet)  : SUCCESS : Traceroute FAIL (too many hops)
Azure Worker -&gt; Master (VPN)       : SUCCESS : Traceroute SUCCESS

Master -&gt; Colo Worker 1 (Internet) : SUCCESS : Traceroute SUCCESS
Master -&gt; Colo Worker 1 (VPN)      : SUCCESS : Traceroute SUCCESS
Colo Worker 1 -&gt; Master (Internet) : SUCCESS : Traceroute FAIL (too many hops)
Colo Worker 1 -&gt; Master (VPN)      : SUCCESS : Traceroute SUCCESS
</code></pre>

<p><strong>Update 5</strong></p>

<p>After running the tests above, it got me thinking about routing and I wondered if it was as simple as providing a route to the controller over the VPN for the service CIDR range (<code>10.96.0.0/12</code>).</p>

<p>So on a host, not included in the cluster, I added a route thus:</p>

<pre><code>route add -net 10.96.0.0/12 gw 192.168.191.100
</code></pre>

<p>And I could then resolve DNS using the <code>kube-dns</code> server address:</p>

<pre><code>nslookup www.google.co.uk 10.96.0.10
</code></pre>

<p>SO I then added a route, as above, to one of the worker nodes and tried the same. But it is blocked and I do not get a response.
Given that I can resolve DNS over the VPN with the appropriate route from a non-kubernetes machine, I can only think that there is an IPTables rule that needs updating or adding.</p>

<p>I think this is almost there, just one last bit to fix.</p>

<p><em>I realise this is wrong as it it the <code>kube-proxy</code> should do the DNS resolution on each host. I am leaving it here for information.</em></p>

<p>Sounds like you are running on AWS. I suspect that your AWS security group is not allowing DNS traffic to go through. You can try allowing all traffic to the Security Group(s) where all your master and nodes are, to see if that's the problem.</p>

<p><a href="https://i.stack.imgur.com/ta227.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/ta227.png" alt="sg"></a></p>

<p>You can also check that all your masters and nodes are allowing routing:</p>

<pre><code>cat /proc/sys/net/ipv4/ip_forward
</code></pre>

<p>If not </p>

<pre><code>echo 1 &gt; /proc/sys/net/ipv4/ip_forward
</code></pre>

<p>Hope it helps.</p>
