<p>I would like to configure custom DNS in CoreDNS (to bypass NAT loopback issue, meaning that within the network, IP are not resolved the same than outside the network).</p>

<p>I tried to modify ConfigMap for CoreDNS with a 'fake' domain just to test, but it does not work.
I am using minik8s</p>

<p>Here the config file of config map coredns:</p>

<pre><code>apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . 8.8.8.8 8.8.4.4
        cache 30
        loop
        reload
        loadbalance
    }
    consul.local:53 {
        errors
        cache 30
        forward . 10.150.0.1
    }
kind: ConfigMap
</code></pre>

<p>Then I try to resolve this address using busy box, but it does not work.</p>

<pre><code>$kubectl exec -ti busybox -- nslookup test.consul.local
&gt; nslookup: can't resolve 'test.consul.local'
command terminated with exit code 1
</code></pre>

<p>Even kubernetes DNS is failing</p>

<pre><code>$ kubectl exec -ti busybox -- nslookup kubernetes.default
nslookup: can't resolve 'kubernetes.default'
command terminated with exit code 1
</code></pre>

Woody
<p>I've reproduced your scenario and it works as intended. </p>

<p>Here I'll describe two different ways to use custom DNS on Kubernetes. The first is in Pod level. You can customize what DNS server your pod will use. This is useful in specific cases where you don't want to change this configuration for all pods.</p>

<p>To achieve this, you need to add some optional fields. To know more about it, please read <a href="https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-s-dns-config" rel="nofollow noreferrer">this</a>.
Example: </p>

<pre><code>kind: Pod
metadata:
  name: busybox-custom
  namespace: default
spec:
  containers:
  - name: busybox
    image: busybox:1.28
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
  dnsPolicy: "None"
  dnsConfig:
    nameservers:
      - 8.8.8.8
    searches:
      - ns1.svc.cluster-domain.example
      - my.dns.search.suffix
    options:
      - name: ndots
        value: "2"
      - name: edns0
  restartPolicy: Always
</code></pre>

<pre><code>$ kubectl exec -ti busybox-custom -- nslookup cnn.com
Server:    8.8.8.8
Address 1: 8.8.8.8 dns.google

Name:      cnn.com
Address 1: 2a04:4e42::323
Address 2: 2a04:4e42:400::323
Address 3: 2a04:4e42:200::323
Address 4: 2a04:4e42:600::323
Address 5: 151.101.65.67
Address 6: 151.101.129.67
Address 7: 151.101.193.67
Address 8: 151.101.1.67
</code></pre>

<pre><code>$ kubectl exec -ti busybox-custom -- nslookup kubernetes.default
Server:    8.8.8.8
Address 1: 8.8.8.8 dns.google

nslookup: can't resolve 'kubernetes.default'
command terminated with exit code 1
</code></pre>

<p>As you can see, this method will create problem to resolve internal DNS names.</p>

<p>The second way to achieve that, is to change the DNS on a Cluster level. This is the way you choose and as you can see. </p>

<pre><code>$ kubectl get cm coredns -n kube-system -o yaml
apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           upstream
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        forward . 8.8.8.8 8.8.4.4
        cache 30
        loop
        reload
        loadbalance
    }
kind: ConfigMap
</code></pre>

<p>As you can see, I don't have <code>consul.local:53</code> entry. </p>

<blockquote>
  <p><a href="https://www.consul.io/" rel="nofollow noreferrer">Consul</a> is a service networking solution to connect and secure services
  across any runtime platform and public or private cloud</p>
</blockquote>

<p>This kind of setup is not common and I don't think you need to include this entry in your setup. <strong>This might be your issue and when I add this entry, I face the same issues you reported.</strong></p>

<pre><code>$ kubectl exec -ti busybox -- nslookup cnn.com
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      cnn.com
Address 1: 2a04:4e42:200::323
Address 2: 2a04:4e42:400::323
Address 3: 2a04:4e42::323
Address 4: 2a04:4e42:600::323
Address 5: 151.101.65.67
Address 6: 151.101.193.67
Address 7: 151.101.1.67
Address 8: 151.101.129.67
</code></pre>

<pre><code>$ kubectl exec -ti busybox -- nslookup kubernetes.default
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes.default
Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local
</code></pre>

<p>Another main problem is that you are debugging DNS using the latest busybox image. I highly recommend you to avoid any version newer than 1.28 as it has come know <a href="https://github.com/docker-library/busybox/issues/48" rel="nofollow noreferrer">problems</a> regarding name resolution. </p>

<p>The best busybox image you can use to troubleshot DNS is <a href="https://raw.githubusercontent.com/kubernetes/website/master/content/en/examples/admin/dns/busybox.yaml" rel="nofollow noreferrer">1.28</a> as <a href="https://stackoverflow.com/users/1288818/oleg-butuzov" title="1,840 reputation">Oleg Butuzov</a> recommended on the comments. </p>
