<p>I use next command to check dns issue in my k8s:</p>
<pre><code>kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml
kubectl exec -i -t dnsutils -- nslookup kubernetes.default
</code></pre>
<p>The nslookup result is:</p>
<pre><code>;; connection timed out; no servers could be reached

command terminated with exit code 1
</code></pre>
<p><strong>dnsutils.yaml:</strong></p>
<pre><code>apiVersion: v1
kind: Pod
metadata:
  name: dnsutils
  namespace: default
spec:
  containers:
  - name: dnsutils
    image: gcr.io/kubernetes-e2e-test-images/dnsutils:1.3
    command:
      - sleep
      - &quot;3600&quot;
    imagePullPolicy: IfNotPresent
  restartPolicy: Always
</code></pre>
<p><strong>NOTE:</strong> it's a machine which default disable all ports, so I ask our IT admin already open the port based on next doc <a href="https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#check-required-ports" rel="nofollow noreferrer">check-required-ports</a>, I'm not sure if this matters.</p>
<p>And use next I could get the pod ip of coredns.</p>
<pre><code>kubectl get pods -n kube-system -o wide | grep core
coredns-7877db9d45-swb6c                                 1/1     Running   0          2m58s   10.244.1.8       node2   &lt;none&gt;           &lt;none&gt;
coredns-7877db9d45-zwc8v                                 1/1     Running   0          2m57s   10.244.0.6       node1   &lt;none&gt;           &lt;none&gt;
</code></pre>
<p>Here, <code>10.244.0.6</code> is my master while <code>10.244.1.8</code> is my working node.</p>
<p>Then if I directly specify coredns pod ip:</p>
<p><strong>master node ok:</strong></p>
<pre><code>kubectl exec -i -t dnsutils -- nslookup kubernetes.default 10.244.0.6
Server:         10.244.0.6
Address:        10.244.0.6#53

Name:   kubernetes.default.svc.cluster.local
Address: 10.96.0.1
</code></pre>
<p><strong>work node not ok:</strong></p>
<pre><code># kubectl exec -i -t dnsutils -- nslookup kubernetes.default 10.244.1.8
;; connection timed out; no servers could be reached

command terminated with exit code 1
</code></pre>
<p>So, the question narrow down to why COREDNS on work node not works? Anything I need to pay attention?</p>
<p><strong>Environment:</strong></p>
<ul>
<li>OS: ubuntu18.04</li>
<li>K8S: v1.21.0</li>
<li>Cluster boot command:</li>
</ul>
<pre><code>kubeadm init --pod-network-cidr=10.244.0.0/16
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
</code></pre>

atline
<p>Finally, I find the root cause, this is hardware firewall issue, see <a href="https://github.com/flannel-io/flannel/blob/master/Documentation/troubleshooting.md#firewalls" rel="nofollow noreferrer">this</a>:</p>
<blockquote>
<p><strong>Firewalls</strong><br/>
When using udp backend, flannel uses UDP port 8285 for sending encapsulated packets.<br/>
When using vxlan backend, kernel uses UDP port 8472 for sending encapsulated packets.<br/>
Make sure that your firewall rules allow this traffic for all hosts participating in the overlay network.<br/>
Make sure that your firewall rules allow traffic from pod network cidr visit your kubernetes master node.</p>
</blockquote>
<ul>
<li>When <code>nslookup client</code> on the same node of <code>dns server</code>, it won't trigger firewall block, so everything is ok.</li>
<li>When <code>nslookup client</code> not on the same node of <code>dns server</code>, it will trigger firewall block, so we can't access dns server.</li>
</ul>
<p>So, after open the ports, everything ok now.</p>
