<p>I've been trying to get over this but I'm out of ideas for now hence I'm posting the question here.</p>
<p>I'm experimenting with the Oracle Cloud Infrastructure (OCI) and I wanted to create a Kubernetes cluster which exposes some service.</p>
<p><strong>The goal is:</strong></p>
<ul>
<li>A running managed Kubernetes cluster (OKE)</li>
<li>2 nodes at least</li>
<li>1 service that's accessible for external parties</li>
</ul>
<p><strong>The infra looks the following:</strong></p>
<ul>
<li>A VCN for the whole thing</li>
<li>A private subnet on 10.0.1.0/24</li>
<li>A public subnet on 10.0.0.0/24</li>
<li>NAT gateway for the private subnet</li>
<li>Internet gateway for the public subnet</li>
<li>Service gateway</li>
<li>The corresponding security lists for both subnets which I won't share right now unless somebody asks for it</li>
<li>A containerengine K8S (OKE) cluster in the VCN with public Kubernetes API enabled</li>
<li>A node pool for the K8S cluster with 2 availability domains and with 2 instances right now. The instances are ARM machines with 1 OCPU and 6GB RAM running Oracle-Linux-7.9-aarch64-2021.12.08-0 images.</li>
<li>A namespace in the K8S cluster (call it staging for now)</li>
<li>A deployment which refers to a custom NextJS application serving traffic on port 3000</li>
</ul>
<p>And now it's the point where I want to expose the service running on port 3000.</p>
<p><strong>I have 2 obvious choices:</strong></p>
<ul>
<li>Create a LoadBalancer service in K8S which will spawn a classic Load Balancer in OCI, set up it's listener and set up the backendset referring to the 2 nodes in the cluster, plus it adjusts the subnet security lists to make sure traffic can flow</li>
<li>Create a Network Load Balancer in OCI and create a NodePort on K8S and manually configure the NLB to the ~same settings as the classic Load Balancer</li>
</ul>
<p>The first one works perfectly fine but I want to use this cluster with minimal costs so I decided to experiment with option 2, the NLB since it's way cheaper (zero cost).</p>
<p>Long story short, everything works and I can access the NextJS app on the IP of the NLB most of the time but sometimes I couldn't. I decided to look it up what's going on and turned out the NodePort that I exposed in the cluster isn't working how I'd imagine.</p>
<p>The service behind the NodePort is only accessible on the Node that's running the pod in K8S. Assume NodeA is running the service and NodeB is just there chilling. If I try to hit the service on NodeA, everything is fine. But when I try to do the same on NodeB, I don't get a response at all.</p>
<p>That's my problem and I couldn't figure out what could be the issue.</p>
<p><strong>What I've tried so far:</strong></p>
<ul>
<li>Switching from ARM machines to AMD ones - no change</li>
<li>Created a bastion host in the public subnet to test which nodes are responding to requests. Turned out only the node responds that's running the pod.</li>
<li>Created a regular LoadBalancer in K8S with the same config as the NodePort (in this case OCI will create a classic Load Balancer), that works perfectly</li>
<li>Tried upgrading to Oracle 8.4 images for the K8S nodes, didn't fix it</li>
<li>Ran the Node Doctor on the nodes, everything is fine</li>
<li>Checked the logs of kube-proxy, kube-flannel, core-dns, no error</li>
<li>Since the cluster consists of 2 nodes, I gave it a try and added one more node and the service was not accessible on the new node either</li>
<li>Recreated the cluster from scratch</li>
</ul>
<p><strong>Edit:</strong> Some update. I've tried to use a DaemonSet instead of a regular Deployment for the pod to ensure that as a temporary solution, all nodes are running at least one instance of the pod and surprise. The node that was previously not responding to requests on that specific port, it still does not, even though a pod is running on it.</p>
<p><strong>Edit2:</strong> Originally I was running the latest K8S version for the cluster (v1.21.5) and I tried downgrading to v1.20.11 and unfortunately the issue is still present.</p>
<p><strong>Edit3:</strong> Checked if the NodePort is open on the node that's not responding and it is, at least kube-proxy is listening on it.</p>
<pre><code>tcp        0      0 0.0.0.0:31600           0.0.0.0:*               LISTEN      16671/kube-proxy
</code></pre>
<p><strong>Edit4:</strong>: Tried adding whitelisting iptables rules but didn't change anything.</p>
<pre><code>[opc@oke-cdvpd5qrofa-nyx7mjtqw4a-svceq4qaiwq-0 ~]$ sudo iptables -P FORWARD ACCEPT
[opc@oke-cdvpd5qrofa-nyx7mjtqw4a-svceq4qaiwq-0 ~]$ sudo iptables -P INPUT ACCEPT
[opc@oke-cdvpd5qrofa-nyx7mjtqw4a-svceq4qaiwq-0 ~]$ sudo iptables -P OUTPUT ACCEPT
</code></pre>
<p><strong>Edit5:</strong> Just as a trial, I created a LoadBalancer once more to verify if I'm gone completely mental and I just didn't notice this error when I tried or it really works. Funny thing, it works perfectly fine through the classic load balancer's IP. <strong>But</strong> when I try to send a request to the nodes directly on the port that was opened for the load balancer (it's 30679 for now). I get response only from the node that's running the pod. From the other, still nothing yet through the load balancer, I get 100% successful responses.</p>
<p>Bonus, here's the iptables from the Node that's not responding to requests, not too sure what to look for:</p>
<pre><code>[opc@oke-cn44eyuqdoq-n3ewna4fqra-sx5p5dalkuq-1 ~]$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
KUBE-NODEPORTS  all  --  anywhere             anywhere             /* kubernetes health check service ports */
KUBE-EXTERNAL-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes externally-visible service portals */
KUBE-FIREWALL  all  --  anywhere             anywhere

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
KUBE-FORWARD  all  --  anywhere             anywhere             /* kubernetes forwarding rules */
KUBE-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes service portals */
KUBE-EXTERNAL-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes externally-visible service portals */
ACCEPT     all  --  10.244.0.0/16        anywhere
ACCEPT     all  --  anywhere             10.244.0.0/16

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
KUBE-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes service portals */
KUBE-FIREWALL  all  --  anywhere             anywhere

Chain KUBE-EXTERNAL-SERVICES (2 references)
target     prot opt source               destination

Chain KUBE-FIREWALL (2 references)
target     prot opt source               destination
DROP       all  --  anywhere             anywhere             /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000
DROP       all  -- !loopback/8           loopback/8           /* block incoming localnet connections */ ! ctstate RELATED,ESTABLISHED,DNAT

Chain KUBE-FORWARD (1 references)
target     prot opt source               destination
DROP       all  --  anywhere             anywhere             ctstate INVALID
ACCEPT     all  --  anywhere             anywhere             /* kubernetes forwarding rules */ mark match 0x4000/0x4000
ACCEPT     all  --  anywhere             anywhere             /* kubernetes forwarding conntrack pod source rule */ ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere             /* kubernetes forwarding conntrack pod destination rule */ ctstate RELATED,ESTABLISHED

Chain KUBE-KUBELET-CANARY (0 references)
target     prot opt source               destination

Chain KUBE-NODEPORTS (1 references)
target     prot opt source               destination

Chain KUBE-PROXY-CANARY (0 references)
target     prot opt source               destination

Chain KUBE-SERVICES (2 references)
target     prot opt source               destination
</code></pre>
<p>Service spec (the running one since it was generated using Terraform):</p>
<pre><code>{
    &quot;apiVersion&quot;: &quot;v1&quot;,
    &quot;kind&quot;: &quot;Service&quot;,
    &quot;metadata&quot;: {
        &quot;creationTimestamp&quot;: &quot;2022-01-28T09:13:33Z&quot;,
        &quot;name&quot;: &quot;web-staging-service&quot;,
        &quot;namespace&quot;: &quot;web-staging&quot;,
        &quot;resourceVersion&quot;: &quot;22542&quot;,
        &quot;uid&quot;: &quot;c092f99b-7c72-4c32-bf27-ccfa1fe92a79&quot;
    },
    &quot;spec&quot;: {
        &quot;clusterIP&quot;: &quot;10.96.99.112&quot;,
        &quot;clusterIPs&quot;: [
            &quot;10.96.99.112&quot;
        ],
        &quot;externalTrafficPolicy&quot;: &quot;Cluster&quot;,
        &quot;ipFamilies&quot;: [
            &quot;IPv4&quot;
        ],
        &quot;ipFamilyPolicy&quot;: &quot;SingleStack&quot;,
        &quot;ports&quot;: [
            {
                &quot;nodePort&quot;: 31600,
                &quot;port&quot;: 3000,
                &quot;protocol&quot;: &quot;TCP&quot;,
                &quot;targetPort&quot;: 3000
            }
        ],
        &quot;selector&quot;: {
            &quot;app&quot;: &quot;frontend&quot;
        },
        &quot;sessionAffinity&quot;: &quot;None&quot;,
        &quot;type&quot;: &quot;NodePort&quot;
    },
    &quot;status&quot;: {
        &quot;loadBalancer&quot;: {}
    }
}
</code></pre>
<p>Any ideas are appreciated.
Thanks guys.</p>

Arnold Galovics
<p>Might not be the ideal fix, but can you try changing the externalTrafficPolicy to Local. This would prevent the health check on the nodes which don't run the application to fail. This way the traffic will only be forwarded to the node where the application is . Setting externalTrafficPolicy to local is also a requirement to preserve source IP of the connection. Also, can you share the health check config for both NLB and LB that you are using. When you change the externalTrafficPolicy, note that the health check for LB would change and the same needs to be applied to NLB.</p>
<p>Edit: Also note that you need a security list/ network security group added to your node subnet/nodepool, which allows traffic on all protocols from the worker node subnet.</p>
