<p>I am running a GKE Cluster with the GCP Load Balancer as my Ingress Controller.</p>
<p>However, I noticed that a specific request to the service hosted in this GKE Cluster was being rejected with and 502 error.</p>
<p>I checked the GCP Loadbalancer logs and I was able to see a return with <code>statusDetails: &quot;failed_to_pick_backend&quot;</code>.</p>
<p>The health field is saying that my backend is healthy. Just to make sure I changed the health check type from HTTP to TCP to see if anything would change but it kept green.</p>
<p>So, what can I be missing if my GCP Loadbalancer is saying that my backend is reachable but at the same time it returns me an failed to pick backend message?</p>
<p>I really appreciate some help on that.</p>
<p>My ingress:</p>
<pre><code>apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-app
  annotations:
    kubernetes.io/ingress.global-static-ip-name: my-app
    networking.gke.io/managed-certificates: my-app-managed-cert
    kubernetes.io/ingress.class: &quot;gce&quot;
    kubernetes.io/ingress.allow-http: &quot;false&quot;
spec:
  defaultBackend:
    service:
      name: my-app-svc
      port:
        number: 3000
</code></pre>
<p>My service:</p>
<pre><code>apiVersion: v1
kind: Service
metadata:
  name: my-app-svc
spec:
  selector:
    app: my-app
  ports:
    - protocol: TCP
      port: 3000
      targetPort: 3000
  type: NodePort
</code></pre>
<p>Running <code>kubectl describe pod &lt;my-pod&gt;</code> I can see</p>
<pre><code>Conditions:
  Type                                       Status
  cloud.google.com/load-balancer-neg-ready   True 
  Initialized                                True 
  Ready                                      True 
  ContainersReady                            True 
  PodScheduled                               True
</code></pre>

<p>Can you verify the timer of your health check? If this is set to 1 second or lower than the health checks that were set to higher value, <code>failed_to_pick_backend</code> error will normally occur.</p>
<p>I recommend you to change the timer to the default value 5 or higher and then test a new deployment. You can check more details about the health check timers on this link[1].</p>
<p>[1] <a href="https://cloud.google.com/load-balancing/docs/health-check-concepts" rel="nofollow noreferrer">https://cloud.google.com/load-balancing/docs/health-check-concepts</a></p>
