<p>I setup a 4 node Kubernetes cluster by following the guide found here: <a href="https://www.tecmint.com/install-a-kubernetes-cluster-on-centos-8/" rel="nofollow noreferrer">https://www.tecmint.com/install-a-kubernetes-cluster-on-centos-8/</a></p>
<p>It has one master and 3 worker nodes.</p>
<p>I'm running a deployment called &quot;hello-world&quot; based on the bashofmann/rancher-demo image with 20 replicas. I've also created a nodeport service called hello-world that maps nodeport 30213 to port 8080 on each respective pod.</p>
<p>See below for the basic details:</p>
<pre><code># kubectl get all
NAME                               READY   STATUS    RESTARTS   AGE
pod/hello-world-655b948488-22dq4   1/1     Running   0          112m
pod/hello-world-655b948488-2fd7f   1/1     Running   0          112m
pod/hello-world-655b948488-2hrtw   1/1     Running   0          112m
pod/hello-world-655b948488-5h4ns   1/1     Running   0          112m
pod/hello-world-655b948488-5zg9w   1/1     Running   0          112m
pod/hello-world-655b948488-7kcsp   1/1     Running   0          112m
pod/hello-world-655b948488-c5m67   1/1     Running   0          112m
pod/hello-world-655b948488-dswcv   1/1     Running   0          112m
pod/hello-world-655b948488-fbtx6   1/1     Running   0          112m
pod/hello-world-655b948488-g7bxp   1/1     Running   0          112m
pod/hello-world-655b948488-gfb4v   1/1     Running   0          112m
pod/hello-world-655b948488-j6lz9   1/1     Running   0          112m
pod/hello-world-655b948488-jthnq   1/1     Running   0          112m
pod/hello-world-655b948488-pm5b8   1/1     Running   0          112m
pod/hello-world-655b948488-qt7gs   1/1     Running   0          112m
pod/hello-world-655b948488-s2hjv   1/1     Running   0          112m
pod/hello-world-655b948488-vcjzz   1/1     Running   0          112m
pod/hello-world-655b948488-vprgn   1/1     Running   0          112m
pod/hello-world-655b948488-x4b9n   1/1     Running   0          112m
pod/hello-world-655b948488-ztfh7   1/1     Running   0          112m

NAME                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
service/hello-world   NodePort    10.110.212.243   &lt;none&gt;        8080:30213/TCP   114m
service/kubernetes    ClusterIP   10.96.0.1        &lt;none&gt;        443/TCP          2d2h

NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/hello-world   20/20   20           20          112m

NAME                                     DESIRED   CURRENT   READY   AGE
replicaset.apps/hello-world-655b948488   20        20        20      112m

# kubectl get nodes -o wide
NAME          STATUS   ROLES                  AGE    VERSION   INTERNAL-IP       EXTERNAL-IP   OS-IMAGE         KERNEL-VERSION                 CONTAINER-RUNTIME
k8s-master    Ready    control-plane,master   2d2h   v1.21.3   192.168.188.190   &lt;none&gt;        CentOS Linux 8   4.18.0-305.10.2.el8_4.x86_64   docker://20.10.7
k8s-worker1   Ready    &lt;none&gt;                 2d2h   v1.21.3   192.168.188.191   &lt;none&gt;        CentOS Linux 8   4.18.0-305.10.2.el8_4.x86_64   docker://20.10.7
k8s-worker2   Ready    &lt;none&gt;                 2d2h   v1.21.3   192.168.188.192   &lt;none&gt;        CentOS Linux 8   4.18.0-305.10.2.el8_4.x86_64   docker://20.10.7
k8s-worker3   Ready    &lt;none&gt;                 2d2h   v1.21.3   192.168.188.193   &lt;none&gt;        CentOS Linux 8   4.18.0-305.10.2.el8_4.x86_64   docker://20.10.7
</code></pre>
<p>I've discovered that the cluster is not load balancing across the three worker nodes.
If I open my web browser and go to <a href="http://192.168.188.191:30213" rel="nofollow noreferrer">http://192.168.188.191:30213</a> then it'll load the website but only when served up by pods on k8s-worker1. Likewise, if I go to <a href="http://192.168.188.192:30213" rel="nofollow noreferrer">http://192.168.188.192:30213</a> then it'll load the website but only when served up by pods on k8s-worker2.</p>
<p>The neat thing about this particular container/pod image is that it'll display which pod is serving up my website request at any given time.  The page refreshes and cycles through all of the available pods in the cluster.  I can see that any time the page successfully refreshes it's only being served by a pod that's present on k8s-worker1.  It'll also display how many replicas are present. I should be seeing 20, but I only see at most 8 replicas.</p>
<p>It'll never load the website served up by any pods on any of the other worker nodes. From the k8s-master I can issue the &quot;curl --insecure <a href="http://192.168.188.191:30213%22" rel="nofollow noreferrer">http://192.168.188.191:30213&quot;</a> command and get a response about 33% of the time.  The rest of the time it fails.  I believe this is because it's trying to load-balance the request to the other worker nodes but those requests fail.</p>
<p>Since I'm still pretty new to this stuff, I'm not sure what to look at.  Is it possible there's something wrong with the replicaset?</p>
<p>Each worker node has the following firewall rules opened up:</p>
<pre><code># firewall-cmd --list-ports
6443/tcp 2379-2380/tcp 10250/tcp 10251/tcp 10252/tcp 10255/tcp 6783/tcp 6783/udp 6784/udp 443/tcp
</code></pre>
<p>Are there more ports I need to open up?  Am I supposed to open up all ports from 3000-32767?  This seems like a possible security vulnerability.</p>

dutsnekcirf
<p>The behaviour that you are facing is probably given by the the type of service that you created for load balancing your pod: You used type <em>NodePort</em>.</p>
<p>Mentioning the <a href="https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types" rel="nofollow noreferrer">official kubernetes documentation</a> the type <em>NodePort</em> is exposing the Service on each Node's IP at a static port (in your case the 30213). In this scenario, by making a request to a node, you are coherently seeing always the same node and the number of pods scheduled on that node, in your mentioned example: 8 (Note that this number may vary with respect to the pods distribution).</p>
<p>If you want to load balance all your pod you should either use a service type: <em>LoadBalancer</em> or <em>ClusterIp</em> + an <a href="https://kubernetes.io/docs/concepts/services-networking/ingress/" rel="nofollow noreferrer">Ingress</a>.</p>
<p>Please, note that both of the options that I mentioned require that the cluster is capable of having an external ip which is externally load balanced &quot;outside&quot;  kubernetes scope. If you are using a managed kubernetes installation (e.g.GKE, EKS, AKS) you will have it &quot;for free&quot;.
Since you are using a custom cluster installation, for the load balanced, you can have a look at <a href="https://metallb.universe.tf/" rel="nofollow noreferrer">MetalLB</a> project.</p>
