<p>I'm struggling with this cross-communication between pods even though clusterIP services are set up for them. All the pods are on the same master node, and in the same namespace. In Summary:</p>
<pre><code>$ kubectl get pods -o wide
NAME                         READY   STATUS    RESTARTS   AGE    IP           NODE          NOMINATED NODE   READINESS GATES
nginx-744f4df6df-rxhph       1/1     Running   0          136m   10.42.0.31   raspberrypi   &lt;none&gt;           &lt;none&gt;
nginx-2-867f4f8859-csn48     1/1     Running   0          134m   10.42.0.32   raspberrypi   &lt;none&gt;           &lt;none&gt;

$ kubectl get svc -o wide
NAME             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE    SELECTOR
nginx-service    ClusterIP   10.43.155.201   &lt;none&gt;        80/TCP                       136m   app=nginx
nginx-service2   ClusterIP   10.43.182.138   &lt;none&gt;        85/TCP                       134m   app=nginx-2
</code></pre>
<p>where I can't <em>curl http://nginx-service2:85</em> from within nginx container, or vice versa... while I validated that this worked from my docker desktop installation:</p>
<pre><code># docker desktop
root@nginx-7dc45fbd74-7prml:/# curl http://nginx-service2:85
&lt;!DOCTYPE html&gt;
&lt;html&gt;
&lt;head&gt;
&lt;title&gt;Welcome to nginx!&lt;/title&gt;
&lt;style&gt;
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
&lt;/style&gt;
&lt;/head&gt;
&lt;body&gt;
&lt;h1&gt;Welcome to nginx!&lt;/h1&gt;
&lt;p&gt;If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.&lt;/p&gt;

&lt;p&gt;For online documentation and support please refer to
&lt;a href=&quot;http://nginx.org/&quot;&gt;nginx.org&lt;/a&gt;.&lt;br/&gt;
Commercial support is available at
&lt;a href=&quot;http://nginx.com/&quot;&gt;nginx.com&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Thank you for using nginx.&lt;/em&gt;&lt;/p&gt;
&lt;/body&gt;
&lt;/html&gt;

# k3s
root@nginx-744f4df6df-rxhph:/# curl http://nginx-service2.pwk3spi-vraptor:85
curl: (6) Could not resolve host: nginx-service2.pwk3spi-vraptor
</code></pre>
<p>After googling the issue (and please correct me if I'm wrong) it seems like a coredns issue, because looking at the logs and see the error timeouts:</p>
<pre><code>$ kubectl get pods -n kube-system
NAME                                     READY   STATUS      RESTARTS   AGE
helm-install-traefik-qr2bd               0/1     Completed   0          153d
metrics-server-7566d596c8-nnzg2          1/1     Running     59         148d
svclb-traefik-kjbbr                      2/2     Running     60         153d
traefik-758cd5fc85-wzjrn                 1/1     Running     20         62d
local-path-provisioner-6d59f47c7-4hvf2   1/1     Running     72         148d
coredns-7944c66d8d-gkdp4                 1/1     Running     0          3m47s

$ kubectl logs coredns-7944c66d8d-gkdp4 -n kube-system
.:53
[INFO] plugin/reload: Running configuration MD5 = 1c648f07b77ab1530deca4234afe0d03
CoreDNS-1.6.9
linux/arm, go1.14.1, 1766568
[ERROR] plugin/errors: 2 1898797220.1916943194. HINFO: read udp 10.42.0.38:50482-&gt;192.168.8.109:53: i/o timeout
[ERROR] plugin/errors: 2 1898797220.1916943194. HINFO: read udp 10.42.0.38:34160-&gt;192.168.8.109:53: i/o timeout
[ERROR] plugin/errors: 2 1898797220.1916943194. HINFO: read udp 10.42.0.38:53485-&gt;192.168.8.109:53: i/o timeout
[ERROR] plugin/errors: 2 1898797220.1916943194. HINFO: read udp 10.42.0.38:46642-&gt;192.168.8.109:53: i/o timeout
[ERROR] plugin/errors: 2 1898797220.1916943194. HINFO: read udp 10.42.0.38:55329-&gt;192.168.8.109:53: i/o timeout
[ERROR] plugin/errors: 2 1898797220.1916943194. HINFO: read udp 10.42.0.38:44471-&gt;192.168.8.109:53: i/o timeout
[ERROR] plugin/errors: 2 1898797220.1916943194. HINFO: read udp 10.42.0.38:49182-&gt;192.168.8.109:53: i/o timeout
[ERROR] plugin/errors: 2 1898797220.1916943194. HINFO: read udp 10.42.0.38:54082-&gt;192.168.8.109:53: i/o timeout
[ERROR] plugin/errors: 2 1898797220.1916943194. HINFO: read udp 10.42.0.38:48151-&gt;192.168.8.109:53: i/o timeout
[ERROR] plugin/errors: 2 1898797220.1916943194. HINFO: read udp 10.42.0.38:48599-&gt;192.168.8.109:53: i/o timeout
</code></pre>
<p>where people recommended</p>
<ul>
<li>changing the coredns config map to forward to your master node IP</li>
</ul>
<blockquote>
<p>... other CoreFile stuff</p>
<p>forward . <em>host server IP</em></p>
<p>... other CoreFile stuff</p>
</blockquote>
<ul>
<li>or adding your coredns clusterip IP as a nameserver to /etc/resolve.conf</li>
</ul>
<blockquote>
<p>search default.svc.cluster.local svc.cluster.local cluster.local</p>
<p>nameserver 10.42.0.38</p>
<p>nameserver 192.168.8.1</p>
<p>nameserver fe80::266:19ff:fea7:85e7%wlan0</p>
</blockquote>
<p>, however didn't find that these solutions worked.</p>
<p>Details for reference:</p>
<pre><code>$ kubectl get nodes -o wide
NAME          STATUS   ROLES    AGE    VERSION        INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION   CONTAINER-RUNTIME
raspberrypi   Ready    master   153d   v1.18.9+k3s1   192.168.8.109   &lt;none&gt;        Raspbian GNU/Linux 10 (buster)   5.10.9-v7l+      containerd://1.3.3-k3s2

$ kubectl get svc -n kube-system -o wide
NAME                 TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                      AGE    SELECTOR
kube-dns             ClusterIP      10.43.0.10      &lt;none&gt;          53/UDP,53/TCP,9153/TCP       153d   k8s-app=kube-dns
metrics-server       ClusterIP      10.43.205.8     &lt;none&gt;          443/TCP                      153d   k8s-app=metrics-server
traefik-prometheus   ClusterIP      10.43.222.138   &lt;none&gt;          9100/TCP                     153d   app=traefik,release=traefik
traefik              LoadBalancer   10.43.249.133   192.168.8.109   80:31222/TCP,443:32509/TCP   153d   app=traefik,release=traefik

$ kubectl get ep kube-dns -n kube-system
NAME       ENDPOINTS                                     AGE
kube-dns   10.42.0.38:53,10.42.0.38:9153,10.42.0.38:53   153d
</code></pre>
<p>No idea where I'm going wrong, or if I focused on the wrong stuff, or how to continue. Any help will be much appreciated, please.</p>

Paul
<p>When all else fails..... go back to the manual. I tried finding the 'issue' in all the wrong places, while I just had to follow Rancher's installation documentation for k3s (sigh).</p>
<p><a href="https://rancher.com/docs/k3s/latest/en/quick-start/" rel="nofollow noreferrer">Rancher's documentation</a> is very good (you just have to actually follow it), where they state that when installing k3s on <strong>Raspbian Buster environments</strong></p>
<blockquote>
<p>check version:</p>
<pre><code>$ lsb_release -a
No LSB modules are available.
Distributor ID: Raspbian
Description:    Raspbian GNU/Linux 10 (buster)
Release:        10
Codename:       buster
</code></pre>
</blockquote>
<p>you need to change to <em>legacy iptables</em>, stating to run (<a href="https://rancher.com/docs/k3s/latest/en/advanced/#enabling-legacy-iptables-on-raspbian-buster" rel="nofollow noreferrer">link</a>):</p>
<pre><code>sudo iptables -F
sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
sudo reboot
</code></pre>
<blockquote>
<p><strong>note</strong> that when setting the iptables, do it directly on the pi, not via ssh. You will be kicked out</p>
</blockquote>
<p>After doing this, all my services were happy, and could curl each other from within the containers via their defined clusterIP service names etc.</p>
