<p>I have a 4 node Kubernetes cluster, 1 x controller and 3 x workers. The following shows how they are configured with the versions.</p>

<p><code>
NAME             STATUS    ROLES     AGE       VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
k8s-ctrl-1       Ready     master    1h        v1.11.2   192.168.191.100       &lt;none&gt;        Ubuntu 18.04.1 LTS   4.15.0-1021-aws     docker://18.6.1
turtle-host-01   Ready     &lt;none&gt;    1h        v1.11.2   192.168.191.53   &lt;none&gt;        Ubuntu 18.04.1 LTS   4.15.0-29-generic   docker://18.6.1
turtle-host-02   Ready     &lt;none&gt;    1h        v1.11.2   192.168.191.2    &lt;none&gt;        Ubuntu 18.04.1 LTS   4.15.0-34-generic   docker://18.6.1
turtle-host-03   Ready     &lt;none&gt;    1h        v1.11.2   192.168.191.3    &lt;none&gt;        Ubuntu 18.04.1 LTS   4.15.0-33-generic   docker://18.6.1
</code></p>

<p>Each of the nodes has two network interfaces, for arguments sake <code>eth0</code> and <code>eth1</code>. <code>eth1</code> is the network that I want to the cluster to work on. I setup the controller using <code>kubeadm init</code> and passed <code>--api-advertise-address 192.168.191.100</code>. The worker nodes where then joined using this address.</p>

<p>Finally on each node I modified the kubelet service to have <code>--node-ip</code> set so that the layout looks as above.</p>

<p>The cluster appears to be working correctly and I can create pods, deployments etc. However the issue I have is that none of the pods are able to use the <code>kube-dns</code> service for DNS resolution.</p>

<p>This is not a problem with resolution, rather that the machines cannot connect to the DNS service to perform the resolution. For example if I run a <code>busybox</code> container and access it to perform <code>nslookup</code> i get the following:</p>

<p><code>
/ # nslookup www.google.co.uk
nslookup: read: Connection refused
nslookup: write to '10.96.0.10': Connection refused
</code></p>

<p>I have a feeling that this is down to not using the default network and because of that I suspect some Iptables rules are not correct, that being said these are just guesses.</p>

<p>I have tried both the Flannel overlay and now Weave net. The pod CIDR range is <code>10.32.0.0/16</code> and the service CIDR is as default.</p>

<p>I have noticed that with Kubernetes 1.11 there are now pods called <code>coredns</code> rather than one <code>kube-dns</code>.</p>

<p>I hope that this is a good place to ask this question. I am sure I am missing something small but vital so if anyone has any ideas that would be most welcome.</p>

<p><strong>Update #1:</strong></p>

<p>I should have said that the nodes are not all in the same place. I have a VPN running between them all and this is the network I want things to communicate over. It is an idea I had to try and have distributed nodes.</p>

<p><strong>Update #2:</strong></p>

<p>I saw another answer on SO (<a href="https://stackoverflow.com/questions/30992961/dns-in-kubernetes-not-working?answertab=active#tab-top">DNS in Kubernetes not working</a>) that suggested <code>kubelet</code> needed to have <code>--cluster-dns</code> and <code>--cluster-domain</code> set. This is indeed the case on my DEV K8s cluster that I have running at home (on one network).</p>

<p>However it is not the case on this cluster and I suspect this is down to a later version. I did add the two settings to all nodes in the cluster, but it did not make things work.</p>

<p><strong>Update #3</strong></p>

<p>The topology of the cluster is as follows.</p>

<ul>
<li>1 x Controller is in AWS</li>
<li>1 x Worker is in Azure</li>
<li>2 x Worker are physical machines in a colo Data Centre</li>
</ul>

<p>All machines are connected to each other using ZeroTier VPN on the 192.168.191.0/24 network.</p>

<p>I have <em>not</em> configured any special routing. I agree that this is probably where the issue is, but I am not 100% sure what this routing should be.</p>

<p>WRT to <code>kube-dns</code> and <code>nginx</code>, I have not tainted my controller so <code>nginx</code> is not on the master, not is <code>busybox</code>. <code>nginx</code> and <code>busybox</code> are on workers 1 and 2 respectively.</p>

<p>I have used <code>netcat</code> to test connection to <code>kube-dns</code> and I get the following:</p>

<p><code>
/ # nc -vv 10.96.0.10 53
nc: 10.96.0.10 (10.96.0.10:53): Connection refused
sent 0, rcvd 0
/ # nc -uvv 10.96.0.10 53
10.96.0.10 (10.96.0.10:53) open
</code></p>

<p>The UDP connection does not complete.</p>

<p>I modified my setup so that I could run containers on the controller, so <code>kube-dns</code>, <code>nginx</code> and <code>busybox</code> are all on the controller, and I am able to connect and resolve DNS queries against 10.96.0.10.</p>

<p>So all this does point to routing or IPTables IMHO, I just need to work out what that should be.</p>

<p><strong>Update #4</strong></p>

<p>In response to comments I can confirm the following ping test results.</p>

<pre><code>Master -&gt; Azure Worker (Internet)  : SUCCESS : Traceroute SUCCESS
Master -&gt; Azure Worker (VPN)       : SUCCESS : Traceroute SUCCESS
Azure Worker -&gt; Master (Internet)  : SUCCESS : Traceroute FAIL (too many hops)
Azure Worker -&gt; Master (VPN)       : SUCCESS : Traceroute SUCCESS

Master -&gt; Colo Worker 1 (Internet) : SUCCESS : Traceroute SUCCESS
Master -&gt; Colo Worker 1 (VPN)      : SUCCESS : Traceroute SUCCESS
Colo Worker 1 -&gt; Master (Internet) : SUCCESS : Traceroute FAIL (too many hops)
Colo Worker 1 -&gt; Master (VPN)      : SUCCESS : Traceroute SUCCESS
</code></pre>

<p><strong>Update 5</strong></p>

<p>After running the tests above, it got me thinking about routing and I wondered if it was as simple as providing a route to the controller over the VPN for the service CIDR range (<code>10.96.0.0/12</code>).</p>

<p>So on a host, not included in the cluster, I added a route thus:</p>

<pre><code>route add -net 10.96.0.0/12 gw 192.168.191.100
</code></pre>

<p>And I could then resolve DNS using the <code>kube-dns</code> server address:</p>

<pre><code>nslookup www.google.co.uk 10.96.0.10
</code></pre>

<p>SO I then added a route, as above, to one of the worker nodes and tried the same. But it is blocked and I do not get a response.
Given that I can resolve DNS over the VPN with the appropriate route from a non-kubernetes machine, I can only think that there is an IPTables rule that needs updating or adding.</p>

<p>I think this is almost there, just one last bit to fix.</p>

<p><em>I realise this is wrong as it it the <code>kube-proxy</code> should do the DNS resolution on each host. I am leaving it here for information.</em></p>

<p>Following the instruction at this <a href="https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/" rel="nofollow noreferrer">page</a>, try to run this:</p>

<pre><code>apiVersion: v1
kind: Pod
metadata:
  namespace: default
  name: dns-example
spec:
  containers:
    - name: test
      image: nginx
  dnsPolicy: "None"
  dnsConfig:
    nameservers:
      - 1.2.3.4
    searches:
      - ns1.svc.cluster.local
      - my.dns.search.suffix
    options:
      - name: ndots
        value: "2"
      - name: edns0
</code></pre>

<p>and see if a manual configuration works or you have some networking DNS problem.</p>
