<p>I'm wondering what steps to take when troubleshooting why a Google load balancer would see Nodes within a cluster as unhealthy?</p>

<p>Using Google Kubernetes, I have a cluster with 3 nodes, all deployments are running readiness and liveness checks. All are reporting that they are healthy.</p>

<p>The load balancer is built from the helm nginx-ingress:</p>

<p><a href="https://github.com/helm/charts/tree/master/stable/nginx-ingress" rel="nofollow noreferrer">https://github.com/helm/charts/tree/master/stable/nginx-ingress</a></p>

<p>It's used as a single ingress for all the deployment applications within the cluster.</p>

<p>Visually scanning the ingress controllers logs:</p>

<pre><code>kubectl logs &lt;ingress-controller-name&gt;
</code></pre>

<p>shows only the usual nginx output <code>... HTTP/1.1" 200 ...</code> I can't see any health checks within these logs. Not sure if I should but nothing to suggest anything is unhealhty.</p>

<p>Running a describe against the ingress controller shows no events, but it does show a liveness and readiness check which I'm not too sure would actually pass:</p>

<pre><code>Name:               umbrella-ingress-controller-****
Namespace:          default
Priority:           0
PriorityClassName:  &lt;none&gt;
Node:               gke-multi-client-n1--2cpu-4ram-****/10.154.0.50
Start Time:         Fri, 15 Nov 2019 21:23:36 +0000
Labels:             app=ingress
                    component=controller
                    pod-template-hash=7c55db4f5c
                    release=umbrella
Annotations:        kubernetes.io/limit-ranger: LimitRanger plugin set: cpu request for container ingress-controller
Status:             Running
IP:                 ****
Controlled By:      ReplicaSet/umbrella-ingress-controller-7c55db4f5c
Containers:
  ingress-controller:
    Container ID:  docker://****
    Image:         quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.24.1
    Image ID:      docker-pullable://quay.io/kubernetes-ingress-controller/nginx-ingress-controller@sha256:****
    Ports:         80/TCP, 443/TCP
    Host Ports:    0/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --default-backend-service=default/umbrella-ingress-default-backend
      --election-id=ingress-controller-leader
      --ingress-class=nginx
      --configmap=default/umbrella-ingress-controller
    State:          Running
      Started:      Fri, 15 Nov 2019 21:24:38 +0000
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:      100m
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       umbrella-ingress-controller-**** (v1:metadata.name)
      POD_NAMESPACE:  default (v1:metadata.namespace)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from umbrella-ingress-token-**** (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  umbrella-ingress-token-2tnm9:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  umbrella-ingress-token-****
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  &lt;none&gt;
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          &lt;none&gt;
</code></pre>

<p>However, using Googles console, I navigate to the load balancers details and can see the following:</p>

<p><a href="https://i.stack.imgur.com/zmY3w.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/zmY3w.png" alt="enter image description here"></a></p>

<p>Above 2 of the nodes seem to be having issues, albeit I can't find the issues.</p>

<p>At this point the load balancer is still serving traffic via the third, healthy node, however it will occasionally drop that and show me the following:</p>

<p><a href="https://i.stack.imgur.com/mwMgG.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/mwMgG.png" alt="enter image description here"></a></p>

<p>At this point no traffic gets past the load balancer, so all the applications on the nodes are unreachable.</p>

<p>Any help with where I should be looking to troubleshoot this would be great.</p>

<p>---- edit 17/11/19</p>

<p>Below is the nginx-ingress config passed via helm:</p>

<pre><code>ingress:
  enabled: true
  rbac.create: true
  controller:
    service:
      externalTrafficPolicy: Local
      loadBalancerIP: ****
  configData:
    proxy-connect-timeout: "15"
    proxy-read-timeout: "600"
    proxy-send-timeout: "600"
    proxy-body-size: "100m"
</code></pre>

<p>This is expected behavior. Using <a href="https://kubernetes.io/docs/tutorials/services/source-ip/#source-ip-for-services-with-type-loadbalancer" rel="nofollow noreferrer"><code>externalTrafficPolicy: local</code></a> configures the service so that only nodes where a serving pod exists will accept traffic. What this means is that any node that does not have a serving pod that receives traffic to the service will drop the packet.</p>

<p>The GCP Network Loadbalancer is still sending traffic to each node to test the health. The health check will use the service NodePort. Any node that contains nginx loadbalancer pods will respond to the health check. Any node that does not have an nginx load balancer pod will drop the packet so the check fails. </p>

<p>This results in only certain nodes showing as healthy.</p>

<p>For the nginx ingress controller, I recommend using the default value of <code>cluster</code> instead of changing it to <code>local</code>.</p>
