<p>Ive hit a wall I'm hoping the SO community can advise on where to go next. I've set up a 6 node kubernetes cluster with calico as the networking service. I've only got two pods, the dns debugging pods from kubernetes and a mysql pod. Well and the kube-system pods.
Anyways, I've been at this all day. I've started from scratch 3 times and I keep hitting a wall when it comes to DNS. I've been trying to sort through why I can't access my pods externally. Here are my configs.</p>
<p><strong>mysql.yaml</strong></p>
<pre><code>kind: Service
metadata:
  name: mysql
  namespace: new_namespace
spec:
  type: ExternalName
  externalName: mysql.new_namespace.svc.cluster.local
  ports:
  - port: 3306
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql
  namespace: new_namespace
spec:
  selector:
    matchLabels:
      app: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - image: mysql:5.6
        name: mysql
        env:
          # Use secret in real usage
        - name: MYSQL_ROOT_PASSWORD
          value: *******
        securityContext:
         runAsUser: 0
         allowPrivilegeEscalation: false
        ports:
        - name: mysql
          containerPort: 3306
          protocol: TCP
        volumeMounts:
        - name: mysql-persistent-storage
          mountPath: /var/lib/mysql
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: gluster-claim
</code></pre>
<p>Along with others, I've been primarily following <a href="https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/" rel="nofollow noreferrer">this guide</a>, but have been unsuccessful in determining my issue. DNS issues just... make no sense to me.</p>
<pre><code>$ kubectl exec -i -t -nnew_namespace dnsutils -- nslookup mysql
Server:     192.168.128.10
Address:    192.168.128.10#53

*** Can't find mysql.new_namespace.svc.cluster.local: No answer
</code></pre>
<p>It seems like things <em>should</em> be working...</p>
<pre><code>$ kubectl exec -i -t -nnew_namespace dnsutils -- nslookup kubernetes.default
Server:     192.168.128.10
Address:    192.168.128.10#53

Name:   kubernetes.default.svc.cluster.local
Address: 192.168.128.1
</code></pre>
<pre><code>$ kubectl get pods --all-namespaces
NAMESPACE     NAME                                       READY   STATUS    RESTARTS      AGE
kube-system   calico-kube-controllers-58497c65d5-mcmb4   1/1     Running   0             31m
kube-system   calico-node-27ln4                          1/1     Running   0             36m
kube-system   calico-node-dngfs                          1/1     Running   6 (39m ago)   45m
kube-system   calico-node-nq6bz                          1/1     Running   0             38m
kube-system   calico-node-p6jwz                          1/1     Running   0             35m
kube-system   calico-node-p8fzr                          1/1     Running   0             35m
kube-system   calico-node-wlzr9                          1/1     Running   0             35m
kube-system   calico-typha-68857595fc-kgnvx              1/1     Running   0             45m
kube-system   calico-typha-68857595fc-n4hhq              1/1     Running   0             45m
kube-system   calico-typha-68857595fc-vjgkc              1/1     Running   0             45m
kube-system   coredns-78fcd69978-25bxh                   1/1     Running   0             26m
kube-system   coredns-78fcd69978-cfl52                   1/1     Running   0             26m
kube-system   etcd-new_namespace-master                         1/1     Running   3             49m
kube-system   kube-apiserver-new_namespace-master               1/1     Running   0             49m
kube-system   kube-controller-manager-new_namespace-master      1/1     Running   0             31m
kube-system   kube-proxy-4zx4m                           1/1     Running   0             35m
kube-system   kube-proxy-hhvh7                           1/1     Running   0             38m
kube-system   kube-proxy-m8sph                           1/1     Running   0             35m
kube-system   kube-proxy-qrfx7                           1/1     Running   0             49m
kube-system   kube-proxy-tkb4m                           1/1     Running   0             35m
kube-system   kube-proxy-vct78                           1/1     Running   0             36m
kube-system   kube-scheduler-new_namespace-master               1/1     Running   3             49m
new_namespace        dnsutils                                   1/1     Running   0             30m
new_namespace        mysql-554fd8859d-hb7lp                     1/1     Running   0             4m5s
</code></pre>
<pre><code>$ kubectl get svc --all-namespaces
NAMESPACE     NAME              TYPE           CLUSTER-IP        EXTERNAL-IP                      PORT(S)                  AGE
default       kubernetes        ClusterIP      192.168.128.1     &lt;none&gt;                           443/TCP                  51m
kube-system   calico-typha      ClusterIP      192.168.239.47    &lt;none&gt;                           5473/TCP                 47m
kube-system   kube-dns          ClusterIP      192.168.128.10    &lt;none&gt;                           53/UDP,53/TCP,9153/TCP   51m
new_namespace        gluster-cluster   ClusterIP      192.168.180.197   &lt;none&gt;                           1/TCP                    30m
new_namespace        mysql             ExternalName   &lt;none&gt;            mysql.new_namespace.svc.cluster.local   &lt;none&gt;                   31m
</code></pre>
<pre><code>$ kubectl get endpoints --all-namespaces
NAMESPACE     NAME              ENDPOINTS                                                      AGE
default       kubernetes        10.1.0.125:6443                                                52m
kube-system   calico-typha      10.1.0.126:5473,10.1.0.127:5473,10.1.0.128:5473                48m
kube-system   kube-dns          192.168.13.1:53,192.168.97.65:53,192.168.13.1:53 + 3 more...   52m
new_namespace        gluster-cluster   10.1.0.125:1,10.1.0.126:1,10.1.0.127:1 + 3 more...             31m
</code></pre>
<pre><code>$ kubectl describe endpoints kube-dns --namespace=kube-system
Name:         kube-dns
Namespace:    kube-system
Labels:       k8s-app=kube-dns
              kubernetes.io/cluster-service=true
              kubernetes.io/name=CoreDNS
Annotations:  endpoints.kubernetes.io/last-change-trigger-time: 2021-09-06T15:30:06Z
Subsets:
  Addresses:          192.168.13.1,192.168.97.65
  NotReadyAddresses:  &lt;none&gt;
  Ports:
    Name     Port  Protocol
    ----     ----  --------
    dns-tcp  53    TCP
    dns      53    UDP
    metrics  9153  TCP

Events:  &lt;none&gt;
</code></pre>
<p>And the logs...don't really mean anything to me. It looks like things are working though? Yet I still can't access mysql..</p>
<pre><code>$ kubectl logs --namespace=kube-system -l k8s-app=kube-dns                                                                                                                                                                                      new_namespace-master: Mon Sep  6 16:01:47 2021

[INFO] 192.168.119.1:52410 - 18128 &quot;A IN mysql.new_namespace.svc.cluster.local. udp 48 false 512&quot; NOERROR qr,aa,rd 97 0.00009327s
[INFO] 192.168.119.1:41837 - 46102 &quot;A IN mysql.new_namespace.new_namespace.svc.cluster.local. udp 55 false 512&quot; NXDOMAIN qr,aa,rd 148 0.000240183s
[INFO] 192.168.119.1:42485 - 36923 &quot;A IN mysql.new_namespace.new_namespace.svc.cluster.local. udp 55 false 512&quot; NXDOMAIN qr,aa,rd 148 0.000097762s
[INFO] 192.168.119.1:54354 - 34171 &quot;A IN mysql.new_namespace.new_namespace.svc.cluster.local. udp 55 false 512&quot; NXDOMAIN qr,aa,rd 148 0.000155781s
[INFO] 192.168.119.1:36491 - 48004 &quot;A IN mysql.new_namespace.svc.cluster.local. udp 48 false 512&quot; NOERROR qr,aa,rd 141 0.000075232s
[INFO] 192.168.119.1:58078 - 26522 &quot;A IN mysql.new_namespace.new_namespace.svc.cluster.local. udp 55 false 512&quot; NXDOMAIN qr,aa,rd 148 0.000096242s
[INFO] 192.168.119.1:59389 - 51728 &quot;AAAA IN mysql.new_namespace.svc.cluster.local. udp 48 false 512&quot; NOERROR qr,aa,rd 141 0.000110561s
[INFO] 192.168.119.1:39553 - 24302 &quot;A IN mysql.new_namespace.new_namespace.svc.cluster.local. udp 55 false 512&quot; NXDOMAIN qr,aa,rd 148 0.000114412s
[INFO] 192.168.119.1:60340 - 28351 &quot;A IN mysql.new_namespace.svc.cluster.local. udp 48 false 512&quot; NOERROR qr,aa,rd 141 0.000175322s
[INFO] 192.168.119.1:36494 - 12725 &quot;A IN kubernetes.default.svc.cluster.local. udp 54 false 512&quot; NOERROR qr,aa,rd 106 0.000149332s
[INFO] 192.168.119.1:45875 - 25210 &quot;AAAA IN mysql. udp 23 false 512&quot; NXDOMAIN qr,rd,ra,ad 98 0.000629398s
[INFO] 192.168.119.1:37467 - 44662 &quot;A IN mysql.new_namespace.svc.cluster.local. udp 48 false 512&quot; NOERROR qr,aa,rd 97 0.000115082s
[INFO] 192.168.119.1:37792 - 59085 &quot;AAAA IN mysql. udp 23 false 512&quot; NXDOMAIN qr,aa,rd,ra 98 0.000043841s
[INFO] 192.168.119.1:47263 - 56267 &quot;AAAA IN mysql.new_namespace.svc.cluster.local. udp 48 false 512&quot; NOERROR qr,aa,rd 141 0.000241662s
[INFO] 192.168.119.1:47070 - 59276 &quot;A IN mysql.new_namespace.svc.cluster.local. udp 48 false 512&quot; NOERROR qr,aa,rd 141 0.000144522s
[INFO] 192.168.119.1:46812 - 32557 &quot;A IN mysql.new_namespace.svc.cluster.local. udp 48 false 512&quot; NOERROR qr,aa,rd 141 0.00008474s
[INFO] 192.168.119.1:57113 - 14895 &quot;AAAA IN mysql.new_namespace.svc.cluster.local. udp 48 false 512&quot; NOERROR qr,aa,rd 141 0.000052781s
[INFO] 192.168.119.1:51403 - 18192 &quot;AAAA IN mysql.new_namespace.svc.cluster.local. udp 48 false 512&quot; NOERROR qr,aa,rd 141 0.000166912s
[INFO] 192.168.119.1:52077 - 43229 &quot;A IN kubernetes.default.new_namespace.svc.cluster.local. udp 61 false 512&quot; NXDOMAIN qr,aa,rd 154 0.000199803s
[INFO] 192.168.119.1:60907 - 16052 &quot;AAAA IN kubernetes.default.svc.cluster.local. udp 54 false 512&quot; NOERROR qr,aa,rd 147 0.000082662s
</code></pre>
<p>Oh and this was the init command I used:</p>
<pre><code>sudo kubeadm reset ; sudo kubeadm init --pod-network-cidr=192.168.0.0/17 --service-cidr=192.168.128.0/17 --apiserver-advertise-address 10.1.0.125  --control-plane-endpoint 10.1.0.125
</code></pre>

<p>Turns out I just hadn't installed Calico correctly. I started from scratch one more time and installed calico using <a href="https://docs.projectcalico.org/getting-started/kubernetes/self-managed-onprem/onpremises" rel="nofollow noreferrer">this guide</a> (NOT THE QUICKSTART), and everything's working. I'm not really sure how I came across finding out calico was installed incorrectly. I think it was because I had two calico controllers in two different namespaces and I was very confused as to why. Anyways, reran the same init command, followed the guide, set up mysql and I have access!</p>
