<p>I have build new Kubernetes cluster <code>v1.20.1</code> single master and single node with Calico CNI.</p>
<p>I deployed the <code>busybox </code>pod in default namespace.</p>
<pre><code># kubectl get pods busybox -o wide
NAME      READY   STATUS    RESTARTS   AGE   IP             NODE        NOMINATED NODE   READINESS GATES
busybox   1/1     Running   0          12m   10.203.0.129   node02   &lt;none&gt;           &lt;none&gt;

 
</code></pre>
<p><strong>nslookup not working</strong></p>
<pre><code>kubectl exec -ti busybox -- nslookup kubernetes.default
Server:    10.96.0.10
Address 1: 10.96.0.10

nslookup: can't resolve 'kubernetes.default'
</code></pre>
<p>cluster is running RHEL 8 with latest update</p>
<p>followed this steps: <a href="https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/" rel="noreferrer">https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/</a></p>
<p><strong>nslookup command not able to reach nameserver</strong></p>
<pre><code># kubectl exec -i -t dnsutils -- nslookup kubernetes.default
;; connection timed out; no servers could be reached

command terminated with exit code 1
</code></pre>
<p><strong>resolve.conf file</strong></p>
<pre><code># kubectl exec -ti dnsutils -- cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local 
nameserver 10.96.0.10
options ndots:5
</code></pre>
<p><strong>DNS pods running</strong></p>
<pre><code># kubectl get pods --namespace=kube-system -l k8s-app=kube-dns
NAME                      READY   STATUS    RESTARTS   AGE
coredns-74ff55c5b-472vx   1/1     Running   1          85m
coredns-74ff55c5b-c75bq   1/1     Running   1          85m
</code></pre>
<p><strong>DNS pod logs</strong></p>
<pre><code> kubectl logs --namespace=kube-system -l k8s-app=kube-dns
.:53
[INFO] plugin/reload: Running configuration MD5 = db32ca3650231d74073ff4cf814959a7
CoreDNS-1.7.0
linux/amd64, go1.14.4, f59c03d
.:53
[INFO] plugin/reload: Running configuration MD5 = db32ca3650231d74073ff4cf814959a7
CoreDNS-1.7.0
linux/amd64, go1.14.4, f59c03d
</code></pre>
<p><strong>Service is defined</strong></p>
<pre><code># kubectl get svc --namespace=kube-system
NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
kube-dns   ClusterIP   10.96.0.10   &lt;none&gt;        53/UDP,53/TCP,9153/TCP   86m

**I can see the endpoints of DNS pod**

# kubectl get endpoints kube-dns --namespace=kube-system
NAME       ENDPOINTS                                               AGE
kube-dns   10.203.0.5:53,10.203.0.6:53,10.203.0.5:53 + 3 more...   86m
</code></pre>
<p><strong>enabled the logging, but didn't see traffic coming to DNS pod</strong></p>
<pre><code># kubectl logs --namespace=kube-system -l k8s-app=kube-dns
.:53
[INFO] plugin/reload: Running configuration MD5 = db32ca3650231d74073ff4cf814959a7
CoreDNS-1.7.0
linux/amd64, go1.14.4, f59c03d
.:53
[INFO] plugin/reload: Running configuration MD5 = db32ca3650231d74073ff4cf814959a7
CoreDNS-1.7.0
linux/amd64, go1.14.4, f59c03d
</code></pre>
<p><strong>I can ping DNS POD</strong></p>
<pre><code># kubectl exec -i -t dnsutils -- ping 10.203.0.5
PING 10.203.0.5 (10.203.0.5): 56 data bytes
64 bytes from 10.203.0.5: seq=0 ttl=62 time=6.024 ms
64 bytes from 10.203.0.5: seq=1 ttl=62 time=6.052 ms
64 bytes from 10.203.0.5: seq=2 ttl=62 time=6.175 ms
64 bytes from 10.203.0.5: seq=3 ttl=62 time=6.000 ms
^C
--- 10.203.0.5 ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 6.000/6.062/6.175 ms
</code></pre>
<p><strong>nmap show port filtered</strong></p>
<pre><code># ke netshoot-6f677d4fdf-5t5cb -- nmap 10.203.0.5
Starting Nmap 7.80 ( https://nmap.org ) at 2021-01-15 22:29 UTC
Nmap scan report for 10.203.0.5
Host is up (0.0060s latency).
Not shown: 997 closed ports
PORT     STATE    SERVICE
53/tcp   filtered domain
8080/tcp filtered http-proxy
8181/tcp filtered intermapper

Nmap done: 1 IP address (1 host up) scanned in 14.33 seconds
</code></pre>
<p><strong>If I schedule the POD on master node, nslookup works nmap show port open?</strong></p>
<pre><code># ke netshoot -- bash
bash-5.0# nslookup kubernetes.default
Server:         10.96.0.10
Address:        10.96.0.10#53

Name:   kubernetes.default.svc.cluster.local
Address: 10.96.0.1

 nmap -p 53 10.96.0.10
Starting Nmap 7.80 ( https://nmap.org ) at 2021-01-15 22:46 UTC
Nmap scan report for kube-dns.kube-system.svc.cluster.local (10.96.0.10)
Host is up (0.000098s latency).

PORT   STATE SERVICE
53/tcp open  domain

Nmap done: 1 IP address (1 host up) scanned in 0.14 seconds
</code></pre>
<p>Why nslookup from POD running on worker node is not working? how to troubleshoot this issue?</p>
<p>I re-build the server two times, still same issue.</p>
<p>Thanks</p>
<p>SR</p>
<h2>Update adding kubeadm config file</h2>
<pre><code># cat kubeadm-config.yaml
---
apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
nodeRegistration:
  criSocket: unix:///run/containerd/containerd.sock
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
  kubeletExtraArgs:
    cgroup-driver: &quot;systemd&quot;
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: stable
controlPlaneEndpoint: &quot;master01:6443&quot;
networking:
  dnsDomain: cluster.local
  podSubnet: 10.0.0.0/14
  serviceSubnet: 10.96.0.0/12
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: &quot;ipvs
</code></pre>
<p>&quot;</p>

<p>First of all, according to the docs - please note that <a href="https://docs.projectcalico.org/getting-started/kubernetes/requirements" rel="noreferrer">Calico</a> and <a href="https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/" rel="noreferrer">kubeadm</a> support <strong>Centos/RHEL 7+</strong>.<br />
In both <code>Calico</code> and <code>kubeadm</code> documentation we can see that they only support <strong>RHEL7+</strong>.</p>
<p>By default <strong>RHEL8</strong> uses <code>nftables</code> instead of <code>iptables</code> ( we can still use <code>iptables</code> but  &quot;iptables&quot; on <strong>RHEL8</strong> is actually using the kernel's nft framework in the background - look at <a href="https://access.redhat.com/discussions/5114921" rel="noreferrer">&quot;Running Iptables on RHEL 8&quot;</a>).</p>
<blockquote>
<p><a href="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/considerations_in_adopting_rhel_8/networking_considerations-in-adopting-rhel-8" rel="noreferrer">9.2.1. nftables replaces iptables as the default network packet filtering framework</a></p>
</blockquote>
<p>I believe that <code>nftables</code> may cause this network issues because as we can find on nftables <a href="https://wiki.nftables.org/wiki-nftables/index.php/Adoption" rel="noreferrer">adoption page</a>:</p>
<blockquote>
<p>Kubernetes does not support nftables yet.</p>
</blockquote>
<p><strong>Note:</strong> For now I highly recommend you to use <strong>RHEL7</strong> instead of <strong>RHEL8</strong>.</p>
<hr />
<p>With that in mind, I'll present some information that may help you with <strong>RHEL8</strong>.<br />
I have reproduced your issue and found a solution that works for me.</p>
<ul>
<li>First I opened ports required by <code>Calico</code> - these ports can be found
<a href="https://docs.projectcalico.org/getting-started/kubernetes/requirements" rel="noreferrer">here</a> under &quot;Network requirements&quot;.<br />
<strong>As workaround:</strong></li>
<li>Next I reverted to the old    <code>iptables</code> backend on all cluster
nodes, you can easily do    so by setting <code>FirewallBackend</code>    in
<code>/etc/firewalld/firewalld.conf</code>    to <code>iptables</code> as described<br />
<a href="https://firewalld.org/2018/07/nftables-backend" rel="noreferrer">here</a>.</li>
<li>Finally I restarted <code>firewalld</code> to make the new rules active.</li>
</ul>
<p>I've tried <code>nslookup</code> from <code>Pod</code> running on worker node  (kworker) and it seems to work correctly.</p>
<pre><code>root@kmaster:~# kubectl get pod,svc -o wide
NAME      READY   STATUS    RESTARTS   AGE    IP           NODE      NOMINATED NODE   READINESS GATES
pod/web   1/1     Running   0          112s   10.99.32.1   kworker   &lt;none&gt;           &lt;none&gt;

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE     SELECTOR
service/kubernetes   ClusterIP   10.99.0.1    &lt;none&gt;        443/TCP   5m51s   &lt;none&gt;
root@kmaster:~# kubectl exec -it web -- bash
root@web:/# nslookup kubernetes.default
Server:         10.99.0.10
Address:        10.99.0.10#53

Name:   kubernetes.default.svc.cluster.local
Address: 10.99.0.1

root@web:/#
</code></pre>
