<p>I can't get Ingress to work on GKE, owing to health check failures. I've tried all of the debugging steps I can think of, including:</p>

<ul>
<li>Verified I'm not running low on any quotas</li>
<li>Verified that my service is accessible from within the cluster</li>
<li>Verified that my service works behind a k8s/GKE Load Balancer. </li>
<li>Verified that <code>healthz</code> checks are passing in Stackdriver logs</li>
</ul>

<p><strong>... I'd love any advice about how to debug or fix. Details below!</strong></p>

<hr>

<p>I have set up a service with type <code>LoadBalancer</code> on GKE. Works great via external IP:</p>

<pre><code>apiVersion: v1
kind: Service
metadata:
  name: echoserver
  namespace: es
spec:
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
  type: LoadBalancer
  selector:
    app: echoserver
</code></pre>

<p>Then I try setting up an Ingress on top of this same service:</p>

<pre><code>apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: echoserver-ingress
  namespace: es
  annotations:
    kubernetes.io/ingress.class: "gce"
    kubernetes.io/ingress.global-static-ip-name: "echoserver-global-ip"
spec:
  backend:
    serviceName: echoserver
    servicePort: 80
</code></pre>

<p>The Ingress gets created, but it thinks the backend nodes are unhealthy:</p>

<pre><code>$ kubectl --namespace es describe ingress echoserver-ingress | grep backends
  backends:     {"k8s-be-31102--&lt;snipped&gt;":"UNHEALTHY"}
</code></pre>

<p>Inspecting the state of the Ingress backend in the GKE web console, I see the same thing:</p>

<p><a href="https://i.stack.imgur.com/DZlIK.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/DZlIK.png" alt="0 of 3 healthy"></a></p>

<p>The health check details appear as expected:</p>

<p><a href="https://i.stack.imgur.com/iwWfv.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/iwWfv.png" alt="health check details"></a></p>

<p>... and from within a pod in my cluster I can call the service successfully:</p>

<pre><code># curl  -vvv echoserver  2&gt;&amp;1 | grep "&lt; HTTP"
&lt; HTTP/1.0 200 OK

# curl  -vvv echoserver/healthz  2&gt;&amp;1 | grep "&lt; HTTP"
&lt; HTTP/1.0 200 OK
</code></pre>

<p>And I can address the service by NodePort:</p>

<pre><code># curl  -vvv 10.0.1.1:31102  2&gt;&amp;1 | grep "&lt; HTTP" 
&lt; HTTP/1.0 200 OK
</code></pre>

<p>(Which goes without saying, because the Load Balancer service I set up in step 1 resulted in a web site that's working just fine.)</p>

<p>I also see <code>healthz</code> checks passing in Stackdriver logs:</p>

<p><a href="https://i.stack.imgur.com/gYZEb.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/gYZEb.png" alt="enter image description here"></a></p>

<p>Regarding quotas, I check and see I'm only using 3 of 30 backend services:</p>

<pre><code>$ gcloud compute project-info describe | grep -A 1 -B 1  BACKEND_SERVICES
- limit: 30.0
  metric: BACKEND_SERVICES
  usage: 3.0
</code></pre>

<p>Had a similar issue a few weeks ago. What fixed it for me was to add a NodePort in the service description so that the Google Cloud Loadbalancer can probe this NodePort. The config that worked for me:</p>

<pre><code>apiVersion: v1
kind: Service
metadata: 
  name: some-service
spec: 
  selector: 
    name: some-app
  type: NodePort
  ports: 
    - port: 80
      targetPort: 8080
      nodePort: 32000
      protocol: TCP
</code></pre>

<p>It might take some time for the ingress to pick this up. You ca re-create the ingress to speed things up.</p>
