<p>I trying to deploy Kubernetes system on my local computer (Ubuntu 18.04) and have some problems with the DNS service (I can't talk with headless services through their DNS name).</p>

<p>I use minikube for running the cluster and the version is - </p>

<pre><code>Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.2", GitCommit:"59603c6e503c87169aea6106f57b9f242f64df89", GitTreeState:"clean", BuildDate:"2020-01-18T23:30:10Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.2", GitCommit:"59603c6e503c87169aea6106f57b9f242f64df89", GitTreeState:"clean", BuildDate:"2020-01-18T23:22:30Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}
</code></pre>

<p>The headless service - </p>

<pre><code>NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)             AGE
kubernetes           ClusterIP   10.96.0.1    &lt;none&gt;        443/TCP             67m
zookeeper-headless   ClusterIP   None         &lt;none&gt;        2888/TCP,3888/TCP   3m58s
</code></pre>

<p>The pods - </p>

<pre><code>NAME                      READY   STATUS    RESTARTS   AGE
zookeeper-statefulset-0   1/1     Running   1          57m
zookeeper-statefulset-1   1/1     Running   1          56m
zookeeper-statefulset-2   1/1     Running   1          54m
</code></pre>

<p>DNS service endpoint which not exists -  (<code>kubectl get ep kube-dns --namespace=kube-system</code>)</p>

<pre><code>NAME       ENDPOINTS   AGE
kube-dns               68m
</code></pre>

<p>The DNS pods (not ready) - </p>

<pre><code>NAME                       READY   STATUS    RESTARTS   AGE
coredns-6955765f44-gv42p   0/1     Running   1          58m
coredns-6955765f44-rfkm2   0/1     Running   1          58m
</code></pre>

<p>and the logs of DNS pod is - </p>

<blockquote>
  <p>[INFO] plugin/reload: Running configuration MD5 =
  4e235fcc3696966e76816bcd9034ebc7 CoreDNS-1.6.5 linux/amd64, go1.13.4,
  c2fd1b2 [INFO] plugin/ready: Still waiting on: "kubernetes" [INFO]
  plugin/ready: Still waiting on: "kubernetes" [INFO] plugin/ready:
  Still waiting on: "kubernetes" E0221 12:50:23.090626       1
  reflector.go:125]
  pkg/mod/k8s.io/client-go@v0.0.0-20190620085101-78d2af792bab/tools/cache/reflector.go:98:
  Failed to list *v1.Endpoints: Get
  <a href="https://10.96.0.1:443/api/v1/endpoints?limit=500&amp;resourceVersion=0" rel="nofollow noreferrer">https://10.96.0.1:443/api/v1/endpoints?limit=500&amp;resourceVersion=0</a>:
  dial tcp 10.96.0.1:443: i/o timeout E0221 12:50:23.090668       1
  reflector.go:125]
  pkg/mod/k8s.io/client-go@v0.0.0-20190620085101-78d2af792bab/tools/cache/reflector.go:98:
  Failed to list *v1.Service: Get
  <a href="https://10.96.0.1:443/api/v1/services?limit=500&amp;resourceVersion=0" rel="nofollow noreferrer">https://10.96.0.1:443/api/v1/services?limit=500&amp;resourceVersion=0</a>:
  dial tcp 10.96.0.1:443: i/o timeout E0221 12:50:23.090671       1
  reflector.go:125]
  pkg/mod/k8s.io/client-go@v0.0.0-20190620085101-78d2af792bab/tools/cache/reflector.go:98:
  Failed to list *v1.Namespace: Get
  <a href="https://10.96.0.1:443/api/v1/namespaces?limit=500&amp;resourceVersion=0" rel="nofollow noreferrer">https://10.96.0.1:443/api/v1/namespaces?limit=500&amp;resourceVersion=0</a>:
  dial tcp 10.96.0.1:443: i/o timeout I0221 12:50:23.090594       1
  trace.go:82] Trace[146678255]: "Reflector
  pkg/mod/k8s.io/client-go@v0.0.0-20190620085101-78d2af792bab/tools/cache/reflector.go:98
  ListAndWatch" (started: 2020-02-21 12:49:53.090061147 +0000 UTC
  m=+0.011664556) (total time: 30.000405618s): Trace[146678255]:
  [30.000405618s] [30.000405618s] END E0221 12:50:23.090626       1
  reflector.go:125]
  pkg/mod/k8s.io/client-go@v0.0.0-20190620085101-78d2af792bab/tools/cache/reflector.go:98:
  Failed to list *v1.Endpoints: Get
  <a href="https://10.96.0.1:443/api/v1/endpoints?limit=500&amp;resourceVersion=0" rel="nofollow noreferrer">https://10.96.0.1:443/api/v1/endpoints?limit=500&amp;resourceVersion=0</a>:
  dial tcp 10.96.0.1:443: i/o timeout E0221 12:50:23.090626       1
  reflector.go:125]
  pkg/mod/k8s.io/client-go@v0.0.0-20190620085101-78d2af792bab/tools/cache/reflector.go:98:
  Failed to list *v1.Endpoints: Get
  <a href="https://10.96.0.1:443/api/v1/endpoints?limit=500&amp;resourceVersion=0" rel="nofollow noreferrer">https://10.96.0.1:443/api/v1/endpoints?limit=500&amp;resourceVersion=0</a>:
  dial tcp 10.96.0.1:443: i/o timeout E0221 12:50:23.090626       1
  reflector.go:125]
  pkg/mod/k8s.io/client-go@v0.0.0-20190620085101-78d2af792bab/tools/cache/reflector.go:98:
  Failed to list *v1.Endpoints: Get
  <a href="https://10.96.0.1:443/api/v1/endpoints?limit=500&amp;resourceVersion=0" rel="nofollow noreferrer">https://10.96.0.1:443/api/v1/endpoints?limit=500&amp;resourceVersion=0</a>:
  dial tcp 10.96.0.1:443: i/o timeout I0221 12:50:23.090644       1
  trace.go:82] Trace[653875127]: "Reflector
  pkg/mod/k8s.io/client-go@v0.0.0-20190620085101-78d2af792bab/tools/cache/reflector.go:98
  ListAndWatch" (started: 2020-02-21 12:49:53.090057185 +0000 UTC
  m=+0.011660587) (total time: 30.00054106s): Trace[653875127]:
  [30.00054106s] [30.00054106s] END I0221 12:50:23.090654       1
  trace.go:82] Trace[1501712764]: "Reflector
  pkg/mod/k8s.io/client-go@v0.0.0-20190620085101-78d2af792bab/tools/cache/reflector.go:98
  ListAndWatch" (started: 2020-02-21 12:49:53.090052023 +0000 UTC
  m=+0.011655434) (total time: 30.000437703s): Trace[1501712764]:
  [30.000437703s] [30.000437703s] END E0221 12:50:23.090668       1
  reflector.go:125]
  pkg/mod/k8s.io/client-go@v0.0.0-20190620085101-78d2af792bab/tools/cache/reflector.go:98:
  Failed to list *v1.Service: Get
  <a href="https://10.96.0.1:443/api/v1/services?limit=500&amp;resourceVersion=0" rel="nofollow noreferrer">https://10.96.0.1:443/api/v1/services?limit=500&amp;resourceVersion=0</a>:
  dial tcp 10.96.0.1:443: i/o timeout E0221 12:50:23.090668       1
  reflector.go:125]
  pkg/mod/k8s.io/client-go@v0.0.0-20190620085101-78d2af792bab/tools/cache/reflector.go:98:
  Failed to list *v1.Service: Get
  <a href="https://10.96.0.1:443/api/v1/services?limit=500&amp;resourceVersion=0" rel="nofollow noreferrer">https://10.96.0.1:443/api/v1/services?limit=500&amp;resourceVersion=0</a>:
  dial tcp 10.96.0.1:443: i/o timeout E0221 12:50:23.090668       1
  reflector.go:125]
  pkg/mod/k8s.io/client-go@v0.0.0-20190620085101-78d2af792bab/tools/cache/reflector.go:98:
  Failed to list *v1.Service: Get
  <a href="https://10.96.0.1:443/api/v1/services?limit=500&amp;resourceVersion=0" rel="nofollow noreferrer">https://10.96.0.1:443/api/v1/services?limit=500&amp;resourceVersion=0</a>:
  dial tcp 10.96.0.1:443: i/o timeout E0221 12:50:23.090671       1
  reflector.go:125]
  pkg/mod/k8s.io/client-go@v0.0.0-20190620085101-78d2af792bab/tools/cache/reflector.go:98:
  Failed to list *v1.Namespace: Get
  <a href="https://10.96.0.1:443/api/v1/namespaces?limit=500&amp;resourceVersion=0" rel="nofollow noreferrer">https://10.96.0.1:443/api/v1/namespaces?limit=500&amp;resourceVersion=0</a>:
  dial tcp 10.96.0.1:443: i/o timeout E0221 12:50:23.090671       1
  reflector.go:125]
  pkg/mod/k8s.io/client-go@v0.0.0-20190620085101-78d2af792bab/tools/cache/reflector.go:98:
  Failed to list *v1.Namespace: Get
  <a href="https://10.96.0.1:443/api/v1/namespaces?limit=500&amp;resourceVersion=0" rel="nofollow noreferrer">https://10.96.0.1:443/api/v1/namespaces?limit=500&amp;resourceVersion=0</a>:
  dial tcp 10.96.0.1:443: i/o timeout E0221 12:50:23.090671       1
  reflector.go:125]
  pkg/mod/k8s.io/client-go@v0.0.0-20190620085101-78d2af792bab/tools/cache/reflector.go:98:
  Failed to list *v1.Namespace: Get
  <a href="https://10.96.0.1:443/api/v1/namespaces?limit=500&amp;resourceVersion=0" rel="nofollow noreferrer">https://10.96.0.1:443/api/v1/namespaces?limit=500&amp;resourceVersion=0</a>:
  dial tcp 10.96.0.1:443: i/o timeout [INFO] plugin/ready: Still waiting
  on: "kubernetes" [INFO] plugin/ready: Still waiting on: "kubernetes"
  [INFO] plugin/ready: Still waiting on: "kubernetes"</p>
</blockquote>

<p>I tried running - </p>

<pre><code>kubectl run -i --tty --image busybox:1.28 dns-test --restart=Never --rm
/ # nslookup headless.default.svc.cluster.local
</code></pre>

<p>and got - </p>

<pre><code>Server:    10.96.0.10
Address 1: 10.96.0.10

nslookup: can't resolve 'headless.default.svc.cluster.local'
</code></pre>

<p>I don't even know where to start to fix that. Can anyone help?</p>

<p><strong>UPDATE</strong></p>

<p>I think I understand what is causing the problem but I do not understand why it is happening.
The problem seems to be happening after the FireWall is activated. For some reason, the core-dns pods can't run and get stuck in a ready state. After I turned off the FireWall by running - 
<code>sudo ufw disable</code>
The core-dns pods state has changed to <code>Running</code> and the service now has endpoint addresses.</p>

<pre><code> kubectl run -i --tty --image busybox:1.28 dns-test --restart=Never --rm
If you don't see a command prompt, try pressing enter.
/ # nslookup  zookeeper-headless.default
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      zookeeper-headless.default
Address 1: 172.17.0.4 zookeeper-statefulset-1.zookeeper-headless.default.svc.cluster.local
Address 2: 172.17.0.5 zookeeper-statefulset-0.zookeeper-headless.default.svc.cluster.local
Address 3: 172.17.0.6 zookeeper-statefulset-2.zookeeper-headless.default.svc.cluster.local
</code></pre>

<p>.</p>

<pre><code>NAME       ENDPOINTS                                               AGE
kube-dns   172.17.0.2:53,172.17.0.3:53,172.17.0.2:53 + 3 more...   34m
</code></pre>

<p>.</p>

<pre><code>NAMESPACE     NAME                             READY   STATUS    RESTARTS           12m
kube-system   coredns-6955765f44-2d8md         1/1     Running   4          34m
kube-system   coredns-6955765f44-n2gcp         1/1     Running   4          34m
</code></pre>

nirkov
<p>I'm thinking I understand what is causing the problem but I do not understand why it is happening.
The problem seems to be happening after the FireWall is activated. For some reason, the core-dns pods can't run and get stuck in a ready state. After I turned off the FireWall by running -</p>

<p><code>sudo ufw disable</code></p>

<p>The core-dns pods state has changed to <code>Running</code> and the service now has endpoint addresses.</p>

<pre><code> kubectl run -i --tty --image busybox:1.28 dns-test --restart=Never --rm
If you don't see a command prompt, try pressing enter.
/ # nslookup  zookeeper-headless.default
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      zookeeper-headless.default
Address 1: 172.17.0.4 zookeeper-statefulset-1.zookeeper-headless.default.svc.cluster.local
Address 2: 172.17.0.5 zookeeper-statefulset-0.zookeeper-headless.default.svc.cluster.local
Address 3: 172.17.0.6 zookeeper-statefulset-2.zookeeper-headless.default.svc.cluster.local
</code></pre>

<p>.</p>

<pre><code>NAME       ENDPOINTS                                               AGE
kube-dns   172.17.0.2:53,172.17.0.3:53,172.17.0.2:53 + 3 more...   34m
</code></pre>

<p>.</p>

<pre><code>NAMESPACE     NAME                             READY   STATUS    RESTARTS           12m
kube-system   coredns-6955765f44-2d8md         1/1     Running   4          34m
kube-system   coredns-6955765f44-n2gcp         1/1     Running   4          34m
</code></pre>
