<p>I attempt to build a Pod that runs a service that requires:</p>
<ol>
<li>cluster-internal services to be resolved and accessed by their FQDN (<code>*.cluster.local</code>),</li>
<li>while also have an active OpenVPN connection to a remote cluster and have services from this remote cluster to be resolved and accessed by their FQDN (<code>*.cluster.remote</code>).</li>
</ol>
<p>The service container within the Pod without an OpenVPN sidecar can access all services provided an FQDN using the <code>*.cluster.local</code> namespace. Here is the <code>/etc/resolv.conf</code> in this case:</p>
<pre><code>nameserver 169.254.25.10
search default.cluster.local svc.cluster.local cluster.local
options ndots:5
</code></pre>
<h4>When OpenVPN sidecar manages <code>resolv.conf</code></h4>
<p>The OpenVPN sidecar is started in the following way:</p>
<pre><code>      containers:
        {{- if .Values.vpn.enabled }}
        - name: vpn
          image: &quot;ghcr.io/wfg/openvpn-client&quot;
          imagePullPolicy: {{ .Values.image.pullPolicy | quote }}
          volumeMounts:
            - name: vpn-working-directory
              mountPath: /data/vpn
          env:
            - name: KILL_SWITCH
              value: &quot;off&quot;
            - name: VPN_CONFIG_FILE
              value: connection.conf
          securityContext:
            privileged: true
            capabilities:
              add:
                - &quot;NET_ADMIN&quot;
          resources:
            limits:
              cpu: 100m
              memory: 80Mi
            requests:
              cpu: 25m
              memory: 20Mi
        {{- end }}
</code></pre>
<p><em>and</em> the OpenVPN client configuration contains the following lines:</p>
<pre><code>        script-security 2
        up /etc/openvpn/up.sh
        down /etc/openvpn/down.sh
</code></pre>
<p>Then OpenVPN client will overwrite <code>resolv.conf</code> so that it contains the following:</p>
<pre><code>nameserver 192.168.255.1
options ndots:5
</code></pre>
<p>In this case, any service in <code>*.cluster.remote</code> is resolved, but no services from <code>*.cluster.local</code>. This is expected.</p>
<h4>When OpenVPN sidecar does not manage <code>resolv.conf</code>, but <code>spec.dnsConfig</code> is provided</h4>
<p>Remove the following lines from the OpenVPN client configuration:</p>
<pre><code>        script-security 2
        up /etc/openvpn/up.sh
        down /etc/openvpn/down.sh
</code></pre>
<p>The <code>spec.dnsConfig</code> is provided as:</p>
<pre><code>
      dnsConfig:
        nameservers:
          - 192.168.255.1
        searches:
          - cluster.remote
</code></pre>
<p>Then, <code>resolv.conf</code> will be the following:</p>
<pre><code>nameserver 192.168.255.1
nameserver 169.254.25.10
search default.cluster.local svc.cluster.local cluster.local  cluster.remote
options ndots:5
</code></pre>
<p>This would work for <code>*.cluster.remote</code>, but not for anything <code>*.cluster.local</code>, because the second nameserver is tried as long as the first times out. I noticed that some folk would get around this limitation by setting up namespace rotation and timeout for 1 second, but this behavior looks very hectic to me, I would not consider this, not even as a workaround. Or maybe I'm missing something. <strong>My first question would be: Could rotation and timeout work in this case?</strong></p>
<p>My second question would be: is there any way to make <code>*.cluster.local</code> and <code>*.cluster.remote</code> DNS resolves work reliably from the service container inside the Pod <em>and</em> without using something like <code>dnsmasq</code>?</p>
<p>My third question would be: if <code>dnsmasq</code> is required, how can I configure it, provided, and overwrite <code>resolv.conf</code> by also making sure that the Kubernetes-provided nameserver can be anything (<code>169.254.25.10</code> in this case).</p>
<p>Best,
Zoltán</p>

<p>I had rather solved the problem by running a sidecar DNS-server, because:</p>
<ul>
<li>it is easier to implement, maintain and understand;</li>
<li>it works without surprises.</li>
</ul>
<p>Here is an example pod with <code>CoreDNS</code>:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: Pod
metadata:
  name: foo
  namespace: default
spec:
  volumes:
  - name: config-volume
    configMap:
      name: foo-config
      items:
        - key: Corefile
          path: Corefile
  dnsPolicy: None # SIgnals Kubernetes that you want to supply your own DNS - otherwise `/etc/resolv.conf` will be overwritten by Kubernetes and there is then no way to update it.
  dnsConfig:
    nameservers:
      - 127.0.0.1 # This will set the local Core DNS as the DNS resolver. When `dnsPolicy` is set, `dnsConfig` must be provided.
  containers:
    - name: dns
      image: coredns/coredns
      env:
        - name: LOCAL_DNS
          value: 10.233.0.3 # insert local DNS IP address (see kube-dns service ClusterIp)
        - name: REMOTE_DNS
          value: 192.168.255.1 # insert remote DNS IP address
      args:
        - '-conf'
        - /etc/coredns/Corefile
      volumeMounts:
        - name: config-volume
          readOnly: true
          mountPath: /etc/coredns
    - name: test
      image: debian:buster
      command:
        - bash
        - -c
        - apt update &amp;&amp; apt install -y dnsutils &amp;&amp; cat /dev/stdout
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: foo-config
  namespace: default
data:
  Corefile: |
    cluster.local:53 {
      errors
      health
      forward . {$LOCAL_DNS}
      cache 30
    }
    cluster.remote:53 {
      errors
      health
      rewrite stop {
        # rewrite cluster.remote to cluster.local and back
        name suffix cluster.remote cluster.local answer auto
      }
      forward . {$REMOTE_DNS}
      cache 30
    }
</code></pre>
<p>The <code>CoreDNS</code> config above simply forwards <code>cluster.local</code> queries to the local service and <code>cluster.remote</code> - to the remote one. Using it I was able to resolve <code>kubernetes</code> service IP of both clusters:</p>
<pre><code>❯ k exec -it -n default foo -c test -- bash
root@foo:/# dig @localhost kubernetes.default.svc.cluster.local +short
10.100.0.1
root@foo:/# dig @localhost kubernetes.default.svc.cluster.remote +short
10.43.0.1
</code></pre>
<p>Update:</p>
<p>Possibly, the following Core DNS configuration is sufficient, in case you require access to the internet as well as <code>cluster.internal</code> is provided by Kubernetes itself:</p>
<pre><code>.:53 {
  errors
  health
  forward . {$LOCAL_DNS}
  cache 30
}
cluster.remote:53 {
  errors
  health
  forward . {$REMOTE_DNS}
  cache 30
}
</code></pre>
