<p>NOTE: I tried to include screenshots but stackoverflow does not allow me to add images with preview so I included them as links.</p>
<p>I deployed a web app on AWS using kOps.
I have two nodes and set up a Network Load Balancer.</p>
<p><a href="https://i.stack.imgur.com/DBOsC.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/DBOsC.png" alt="enter image description here" /></a>
The target group of the NLB has two nodes (each node is an instance made from the same template).</p>
<p><a href="https://i.stack.imgur.com/0jwuk.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/0jwuk.png" alt="enter image description here" /></a>
Load balancer actually seems to be working after checking ingress-nginx-controller logs.
The requests are being distributed over pods correctly. And I can access the service via ingress external address.
But when I go to AWS Console / Target Group, one of the two nodes is marked as  and I am concerned with that.</p>
<p>Nodes are running correctly.
<a href="https://i.stack.imgur.com/cUBxa.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/cUBxa.png" alt="enter image description here" /></a></p>
<p>I tried to execute sh into nginx-controller and tried curl to both nodes with their internal IP address.
For the healthy node, I get nginx response and for the unhealthy node, it times out.
I do not know how nginx was installed on one of the nodes and not on the other one.</p>
<p>Could anybody let me know the possible reasons?</p>

<p>I had exactly the same problem before and this should be documented somewhere on AWS or Kubernetes. The answer is copied from <a href="https://aws.amazon.com/premiumsupport/knowledge-center/eks-unhealthy-worker-node-nginx/" rel="nofollow noreferrer">AWS Premium Support</a></p>
<h3>Short description</h3>
<p>The NGINX Ingress Controller sets the <code>spec.externalTrafficPolicy</code> option to <code>Local</code> to preserve the client IP. Also, requests aren't routed to unhealthy worker nodes. The following troubleshooting implies that you don't need to maintain the cluster IP address or preserve the client IP address.</p>
<h3>Resolution</h3>
<p>If you check the ingress controller service you will see the <code>External Traffic Policy</code> field set to <code>Local</code>.</p>
<pre><code>$ kubectl -n ingress-nginx describe svc ingress-nginx-controller

Output:

Name:                     ingress-nginx-controller
Namespace:                ingress-nginx
...
External Traffic Policy:  Local
...
</code></pre>
<p>This Local setting drops packets that are sent to Kubernetes nodes that aren't running instances of the NGINX Ingress Controller. Assign NGINX pods (from the Kubernetes website) to the nodes that you want to schedule the NGINX Ingress Controller on.</p>
<p>Update the pec.externalTrafficPolicy option to <code>Cluster</code></p>
<pre><code>$ kubectl -n ingress-nginx patch service ingress-nginx-controller -p '{&quot;spec&quot;:{&quot;externalTrafficPolicy&quot;:&quot;Cluster&quot;}}'

Output:
service/ingress-nginx-controller patched
</code></pre>
<p>By default, NodePort services perform source address translation (from the Kubernetes website). For NGINX, this means that the source IP of an HTTP request is always the IP address of the Kubernetes node that received the request. If you set a NodePort to the value of the externalTrafficPolicy field in the ingress-nginx service specification to Cluster, then you can't maintain the source IP address.</p>
