<p>I spinned a <a href="https://github.com/kylemanna/docker-openvpn"><code>docker-openvpn</code></a> container in my (local) <strong>Kubernetes</strong> cluster to access my Services securely and debug <em>dependent services</em> locally.</p>

<p>I can connect to the cluster via the <strong>openVPN</strong> server. However I can't resolve my <strong>Services</strong> via <strong>DNS</strong>.</p>

<p>I managed to get to the point where after setting routes on the VPN server:</p>

<ul>
<li>I can ping a <strong>Pod</strong> <em>by IP</em> (<code>subnet 10.2.0.0/16</code>)</li>
<li>I can ping a <strong>Service</strong> <em>by IP</em> (<code>subnet 10.3.0.0/16</code> like the DNS which is at <code>10.3.0.10</code>)</li>
<li>I can <code>curl</code> to a <strong>Services</strong> <em>by IP</em> and get the data I need.</li>
</ul>

<p>but when i <code>nslookup kubernetes</code> or any <strong>Service</strong>, I get:</p>

<pre><code>nslookup kubernetes
;; Got recursion not available from 10.3.0.10, trying next server
;; Got SERVFAIL reply from 10.3.0.10, trying next server
</code></pre>

<p>I am still missing something for the data to return from the DNS server, but can't figure what I need to do.</p>

<p>How do I debug this <code>SERVFAIL</code> issue in <strong>Kubernetes DNS</strong>?</p>

<p><strong>EDIT:</strong></p>

<p>Things I have noticed and am looking to understand:</p>

<ul>
<li><code>nslookup</code> works to resolve Service name in any pod except the openvpn Pod</li>
<li>while <code>nslookup</code> works in those other Pods, <code>ping</code> does not.</li>
<li>similarly <code>traceroute</code> in those other Pods leads to the flannel layer <code>10.0.2.2</code> and then stops there.</li>
</ul>

<p>from this I guess ICMP must be blocked at the flannel layer, and that doesn't help me figure where DNS is blocked.</p>

<p><strong>EDIT2:</strong></p>

<p>I finally figured how to get nslookup to work: I had to push the DNS search domain to the client with </p>

<pre><code>push "dhcp-option DOMAIN-SEARCH cluster.local"
push "dhcp-option DOMAIN-SEARCH svc.cluster.local"
push "dhcp-option DOMAIN-SEARCH default.svc.cluster.local"
</code></pre>

<p>add with the <code>-p</code> option in the <code>docker-openvpn</code> image</p>

<p>so i end up with</p>

<pre><code>docker run -v /etc/openvpn:/etc/openvpn --rm kylemanna/openvpn ovpn_genconfig \
-u udp://192.168.10.152:1194 \
-n 10.3.0.10 \
-n 192.168.10.1 \
-n 8.8.8.8 \
-n 75.75.75.75 \
-n 75.75.75.76 \
-s 10.8.0.0/24 \
-d \
-p "route 10.2.0.0 255.255.0.0" \
-p "route 10.3.0.0 255.255.0.0" \
-p "dhcp-option DOMAIN cluster.local" \
-p "dhcp-option DOMAIN-SEARCH svc.cluster.local" \
-p "dhcp-option DOMAIN-SEARCH default.svc.cluster.local" 
</code></pre>

<p>Now, <code>nslookup</code> works but <strong><code>curl</code> still does not</strong> </p>

<p>finally my config looks like this:</p>

<pre><code>docker run -v /etc/openvpn:/etc/openvpn --rm kylemanna/openvpn ovpn_genconfig \
-u udp://192.168.10.152:1194 \
-n 10.3.0.10 \
-n 192.168.10.1 \
-n 8.8.8.8 \
-n 75.75.75.75 \
-n 75.75.75.76 \
-s 10.8.0.0/24 \
-N \
-p "route 10.2.0.0 255.255.0.0" \
-p "route 10.3.0.0 255.255.0.0" \
-p "dhcp-option DOMAIN-SEARCH cluster.local" \
-p "dhcp-option DOMAIN-SEARCH svc.cluster.local" \
-p "dhcp-option DOMAIN-SEARCH default.svc.cluster.local"
</code></pre>

<p><code>-u</code> for the VPN server address and port</p>

<p><code>-n</code> for all the DNS servers to use</p>

<p><code>-s</code> to define the VPN subnet (as it defaults to 10.2.0.0 which is used by Kubernetes already)</p>

<p><code>-d</code> to disable NAT</p>

<p><code>-p</code> to push options to the client</p>

<p><code>-N</code> to enable NAT: it seems critical for this setup on Kubernetes</p>

<p>the last part, pushing the search domains to the client, was the key to getting <code>nslookup</code> etc.. to work.</p>

<p>note that curl didn't work at first, but seems to start working after a few seconds. So it does work but it takes a bit of time for curl to be able to resolve.</p>
