<p>for days I'm trying to track down a weird behaviour concerning NodePort Services when running Kubernetes on openSUSE Leap 15.3.</p>
<p>For testing purposes on my own server I installed 3 VMs with openSUSE 15.3. With this article: <a href="https://stackoverflow.com/questions/62795930/how-to-install-kubernetes-in-suse-linux-enterprize-server-15-virtual-machines">How to install kubernetes in Suse Linux enterprize server 15 virtual machines?</a> I set up this Kubernetes Cluster:</p>
<pre><code>kubix01:~ # k get nodes -o wide
NAME      STATUS   ROLES                  AGE   VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION         CONTAINER-RUNTIME
kubix01   Ready    control-plane,master   25h   v1.22.2   192.168.42.51   &lt;none&gt;        openSUSE Leap 15.3   5.3.18-59.27-default   docker://20.10.6-ce
kubix02   Ready    &lt;none&gt;                 25h   v1.22.2   192.168.42.52   &lt;none&gt;        openSUSE Leap 15.3   5.3.18-59.27-default   docker://20.10.6-ce
kubix03   Ready    &lt;none&gt;                 25h   v1.22.2   192.168.42.53   &lt;none&gt;        openSUSE Leap 15.3   5.3.18-59.27-default   docker://20.10.6-ce
</code></pre>
<p>For testing things out I made a new 3 Replica Deployment for a traefik/whoami Image with this yaml:</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: whoami
  labels:
    app: whoami
spec:
  replicas: 3
  selector:
    matchLabels:
      app: whoami
  template:
    metadata:
      labels:
        app: whoami
    spec:
      containers:
      - name: whoami
        image: traefik/whoami
        ports:
        - containerPort: 80
</code></pre>
<p>This results in three Pods spread over the 2 worker nodes as expected:</p>
<pre><code>kubix01:~/k8s/whoami # k get pods -o wide
NAME                      READY   STATUS    RESTARTS      AGE   IP           NODE      NOMINATED NODE   READINESS GATES
whoami-8557b59f65-2qkvq   1/1     Running   2 (24h ago)   25h   10.244.2.7   kubix03   &lt;none&gt;           &lt;none&gt;
whoami-8557b59f65-4wnmd   1/1     Running   2 (24h ago)   25h   10.244.1.6   kubix02   &lt;none&gt;           &lt;none&gt;
whoami-8557b59f65-xhx5x   1/1     Running   2 (24h ago)   25h   10.244.1.7   kubix02   &lt;none&gt;           &lt;none&gt;
</code></pre>
<p>After that I created a NodePort Service for making things available to the world with this yaml:</p>
<pre><code>apiVersion: v1
kind: Service
metadata:
  name: whoami
spec:
  type: NodePort
  selector:
    app: whoami
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 80
      nodePort: 30080
</code></pre>
<p>This is the result:</p>
<pre><code>kubix01:~/k8s/whoami # k get svc
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
kubernetes   ClusterIP   10.96.0.1       &lt;none&gt;        443/TCP          25h
whoami       NodePort    10.105.214.86   &lt;none&gt;        8080:30080/TCP   25h

kubix01:~/k8s/whoami # k describe svc whoami
Name:                     whoami
Namespace:                default
Labels:                   &lt;none&gt;
Annotations:              &lt;none&gt;
Selector:                 app=whoami
Type:                     NodePort
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.105.214.86
IPs:                      10.105.214.86
Port:                     &lt;unset&gt;  8080/TCP
TargetPort:               80/TCP
NodePort:                 &lt;unset&gt;  30080/TCP
Endpoints:                10.244.1.6:80,10.244.1.7:80,10.244.2.7:80
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   &lt;none&gt;
</code></pre>
<p>So everything looks fine and I tested things out with curl:</p>
<ol>
<li>curl on one Cluster Node to PodIP:PodPort</li>
</ol>
<pre><code>kubix01:~/k8s/whoami # curl 10.244.1.6
Hostname: whoami-8557b59f65-4wnmd
IP: 127.0.0.1
IP: 10.244.1.6
RemoteAddr: 10.244.0.0:50380
GET / HTTP/1.1
Host: 10.244.1.6
User-Agent: curl/7.66.0
Accept: */*

kubix01:~/k8s/whoami # curl 10.244.1.7
Hostname: whoami-8557b59f65-xhx5x
IP: 127.0.0.1
IP: 10.244.1.7
RemoteAddr: 10.244.0.0:36062
GET / HTTP/1.1
Host: 10.244.1.7
User-Agent: curl/7.66.0
Accept: */*

kubix01:~/k8s/whoami # curl 10.244.2.7
Hostname: whoami-8557b59f65-2qkvq
IP: 127.0.0.1
IP: 10.244.2.7
RemoteAddr: 10.244.0.0:43924
GET / HTTP/1.1
Host: 10.244.2.7
User-Agent: curl/7.66.0
Accept: */*
</code></pre>
<p>==&gt; Everything works as expected</p>
<ol start="2">
<li>curl on Cluster Node to services ClusterIP:ClusterPort:</li>
</ol>
<pre><code>kubix01:~/k8s/whoami # curl 10.105.214.86:8080
Hostname: whoami-8557b59f65-xhx5x
IP: 127.0.0.1
IP: 10.244.1.7
RemoteAddr: 10.244.0.0:1106
GET / HTTP/1.1
Host: 10.105.214.86:8080
User-Agent: curl/7.66.0
Accept: */*

kubix01:~/k8s/whoami # curl 10.105.214.86:8080
Hostname: whoami-8557b59f65-4wnmd
IP: 127.0.0.1
IP: 10.244.1.6
RemoteAddr: 10.244.0.0:9707
GET / HTTP/1.1
Host: 10.105.214.86:8080
User-Agent: curl/7.66.0
Accept: */*

kubix01:~/k8s/whoami # curl 10.105.214.86:8080
Hostname: whoami-8557b59f65-2qkvq
IP: 127.0.0.1
IP: 10.244.2.7
RemoteAddr: 10.244.0.0:25577
GET / HTTP/1.1
Host: 10.105.214.86:8080
User-Agent: curl/7.66.0
Accept: */*
</code></pre>
<p>==&gt; Everything fine, Traffic is LoadBalanced to the different pods.</p>
<ol start="3">
<li>curl on Cluster Node to NodeIP:NodePort</li>
</ol>
<pre><code>kubix01:~/k8s/whoami # curl 192.168.42.51:30080
Hostname: whoami-8557b59f65-2qkvq
IP: 127.0.0.1
IP: 10.244.2.7
RemoteAddr: 10.244.0.0:5463
GET / HTTP/1.1
Host: 192.168.42.51:30080
User-Agent: curl/7.66.0
Accept: */*

kubix01:~/k8s/whoami # curl 192.168.42.52:30080
^C [NoAnswer]
kubix01:~/k8s/whoami # curl 192.168.42.53:30080
^C [NoAnswer]
</code></pre>
<p>==&gt; NodePort Service is only working at the same Node, no answer from the other nodes</p>
<ol start="4">
<li>curl from another Network Host to NodeIP:NodePort</li>
</ol>
<pre><code>user@otherhost:~$ curl 192.168.42.51:30080
^C [NoAnswer]
user@otherhost:~$ curl 192.168.42.52:30080
^C [NoAnswer]
user@otherhost:~$ curl 192.168.42.53:30080
^C [NoAnswer]
</code></pre>
<p>==&gt; Service is not reachable from the outside at all, no answer on all nodes</p>
<p>Has anybody an idea what is going wrong here?</p>
<p>Thx in advance</p>
<p>T0mcat</p>
<p>PS:
Additionally here a little image for clearing things a bit more. Red curved arrows are the non - working connections, gree curved arrows the working ones:</p>
<p><a href="https://i.stack.imgur.com/L95E7.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/L95E7.png" alt="diagram" /></a></p>

<p>FINALLY found the solution:</p>
<p>Weird behaviour of SUSE concerning ip_forward sysctl setting. During OS Installation the Option &quot;IPv4 forwarding&quot; was activated in the installer. After OS was installed, additionally added &quot;net.ipv4.ip_forward = 1&quot; and
&quot;net.ipv4.conf.all.forwarding = 1&quot; in /etc/sysctl.conf.
With &quot;sysctl -l|grep ip_forward&quot; checked, both options where activated but this obviously wasn't the truth.
Found the file &quot;/etc/sysctl.d/70-yast.conf&quot;... here both options where set to 0, which obviously was the truth. But manually setting these options to 1 in this file still weren't enough...
Using SUSEs YAST tool and entering &quot;System/Network Settings&quot; -&gt; Routing one can see the Option &quot;Enable IPv4 Forwarding&quot;, which was checked (remember: in 70-yast.conf these options where set to 0). After playing a bit with the options in YAST, so that the 70-yast.conf gets rewritten and both options where set to 1 in 70-yast.conf, NodePort services are working as expected now.</p>
<p>Conclusion:</p>
<p>IMHO this seems to be a Bug in SUSE... Activating &quot;IPv4 Forwarding&quot; during Installation leads to a 70-yast.conf with disabled forwarding options. And as a plus, &quot;sysctl -l&quot; displays wrong settings, so as YAST does...
The solution is to play around in YASTs network Settings, so that 70-yast.conf gets rewritten with activated ip_forwarding options.</p>
