<p>Something wrong happend with my RPi 4 cluster based on k3sup.</p>
<p>Everything works as expected until yesterday when I had to reinstall master node operating system. For example, I have a redis installed on master node and then some pods on worker nodes. My pods can not connect to redis via DNS: <code>redis-master.database.svc.cluster.local</code> (but they do day before).</p>
<p>It throws an error that can not resolve domain when I test with busybox like:</p>
<pre><code>kubectl run -it --rm --restart=Never busybox --image=busybox:1.28 -- nslookup redis-master.database.svc.cluster.local
</code></pre>
<p>When I want to ping my service with IP (also on busybox):</p>
<pre><code>kubectl run -it --rm --restart=Never busybox --image=busybox:1.28 -- ping 10.43.115.159
</code></pre>
<p>It shows that 100% packet loss.</p>
<p>I'm able to resolve issue with DNS by simply replace coredns config (replace line with <code>forward . /etc/resolv.conf</code> to <code>forward . 192.168.1.101</code>) but I don't think that's good solution, as earlier I didn't have to do that.</p>
<p>Also, it solves issue for mapping domain to IP, but still connection via IP doesn't work.</p>
<p>My nodes:</p>
<pre><code>NAME     STATUS   ROLES    AGE   VERSION         INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                       KERNEL-VERSION   CONTAINER-RUNTIME
node-4   Ready    &lt;none&gt;   10h   v1.19.15+k3s2   192.168.1.105   &lt;none&gt;        Debian GNU/Linux 10 (buster)   5.10.60-v8+      containerd://1.4.11-k3s1
node-3   Ready    &lt;none&gt;   10h   v1.19.15+k3s2   192.168.1.104   &lt;none&gt;        Debian GNU/Linux 10 (buster)   5.10.60-v8+      containerd://1.4.11-k3s1
node-1   Ready    &lt;none&gt;   10h   v1.19.15+k3s2   192.168.1.102   &lt;none&gt;        Debian GNU/Linux 10 (buster)   5.10.60-v8+      containerd://1.4.11-k3s1
node-0   Ready    master   10h   v1.19.15+k3s2   192.168.1.101   &lt;none&gt;        Debian GNU/Linux 10 (buster)   5.10.63-v8+      containerd://1.4.11-k3s1
node-2   Ready    &lt;none&gt;   10h   v1.19.15+k3s2   192.168.1.103   &lt;none&gt;        Debian GNU/Linux 10 (buster)   5.10.60-v8+      containerd://1.4.11-k3s1
</code></pre>
<p>Master node has a taint: <code>role=master:NoSchedule</code>.</p>
<p>Any ideas?</p>
<p><strong>UPDATE 1</strong></p>
<p>I'm able to connect into redis pod. /etc/resolv.conf from <code>redis-master-0</code></p>
<pre><code>search database.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.43.0.10
options ndots:5
</code></pre>
<p>All services on kubernetes:</p>
<pre><code>NAMESPACE       NAME                    TYPE           CLUSTER-IP      EXTERNAL-IP                                               PORT(S)                      AGE
default         kubernetes              ClusterIP      10.43.0.1       &lt;none&gt;                                                    443/TCP                      6d9h
kube-system     traefik-prometheus      ClusterIP      10.43.94.137    &lt;none&gt;                                                    9100/TCP                     6d8h
registry        proxy-docker-registry   ClusterIP      10.43.16.139    &lt;none&gt;                                                    5000/TCP                     6d8h
kube-system     kube-dns                ClusterIP      10.43.0.10      &lt;none&gt;                                                    53/UDP,53/TCP,9153/TCP       6d9h
kube-system     metrics-server          ClusterIP      10.43.101.30    &lt;none&gt;                                                    443/TCP                      6d9h
database        redis-headless          ClusterIP      None            &lt;none&gt;                                                    6379/TCP                     5d19h
database        redis-master            ClusterIP      10.43.115.159   &lt;none&gt;                                                    6379/TCP                     5d19h
kube-system     traefik                 LoadBalancer   10.43.221.89    192.168.1.102,192.168.1.103,192.168.1.104,192.168.1.105   80:30446/TCP,443:32443/TCP   6d8h
</code></pre>

<p>There was one more thing that was not mentioned. I'm using OpenVPN with NordVPN server list on master node, and use a privoxy for worker nodes.</p>
<p>When you install and run OpenVPN before running kubernetes master, OpenVPN add rules that block kubernetes networking. So, coredns does not work and you can't reach any pod via IP as well.</p>
<p>I'm using RPi 4 cluster, so for me it was good enough to just re-install master node, install kubernetes at first and then configure openvpn. Now everything is working as expected.</p>
<p>It's good enough to order your system units by adding <code>After</code> or <code>Before</code> in service definition. I have VPN systemd service that looks like below:</p>
<pre><code>[Unit]
Description=Enable VPN for System
After=network.target
After=k3s.service

[Service]
Type=simple
ExecStart=/etc/openvpn/start-nordvpn-server.sh

[Install]
WantedBy=multi-user.target
</code></pre>
<p>It guarantee that VPN will be run after kubernetes.</p>
