<p>In our GKE we have one service called <code>php-services</code>. It is defined like so:</p>

<pre><code>apiVersion: v1
kind: Service
metadata:
  name: php-services
  labels:
    name: php-services
spec:
  type: NodePort
  ports:
  - port: 80
  selector:
    name: php-services
</code></pre>

<p>I can access this service from inside the cluster. If I run these commands on one of our pods (in <code>Default</code> namespace), I get expected results:</p>

<pre><code>bash-4.4$ nslookup 'php-services'
   Name:      php-services
   Address 1: 10.15.250.136 php-services.default.svc.cluster.local
</code></pre>

<p>and</p>

<pre><code>bash-4.4$ wget -q -O- 'php-services/health'
   {"status":"ok"}
</code></pre>

<p>So the service is ready and responding correctly. I need to expose this service to foreign traffic. I'm trying to do it with Ingress with following config:</p>

<pre><code>apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-tls
  annotations:
    kubernetes.io/ingress.class: "gce"
    kubernetes.io/tls-acme: "true"
    kubernetes.io/ingress.global-static-ip-name: "kubernetes-ingress"
    kubernetes.io/ingress.allow-http: "false"
    external-dns.alpha.kubernetes.io/hostname: "gke-ingress.goout.net"
  namespace: default
spec:
  tls:
  - hosts:
     - php.service.goout.net
    secretName: router-tls
  rules:
  - host: php.service.goout.net
    http:
      paths:
      - backend:
          serviceName: php-services
          servicePort: 80
        path: /*
</code></pre>

<p>But then accessing <a href="http://php.service.goout.net/health" rel="noreferrer">http://php.service.goout.net/health</a> gives an 502 error:</p>

<blockquote>
  <p>Error: Server Error The server encountered a temporary error and could<br>
  not complete your request.<br>
  Please try again in 30 seconds.  </p>
</blockquote>

<p>We also have other services with the same config which run ok and are accessible form outside. </p>

<p>I've found a <a href="https://stackoverflow.com/questions/49076629/ingress-backend-rest-error-the-server-encountered-a-temporary-error-and-could-n">similar question</a> but that doesn't bring any sufficient answer either.<br>
I've been also following the <a href="https://kubernetes.io/docs/tasks/debug-application-cluster/debug-service/" rel="noreferrer">Debug Service</a> article but that also didn't help as the service itself is OK.</p>

<p>Any help with this issue highly appreciated. </p>

<h2>EDIT TLDR</h2>

<p>GKE Loadbalancer <a href="https://cloud.google.com/load-balancing/docs/health-check-concepts" rel="noreferrer">only accepts HTTP status 200</a> while Kubernetes health checks accept <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/" rel="noreferrer">any code greater than or equal to 200 and less than 400</a>.</p>

<h2>Orignal answer</h2>

<p>Ok, so we've figured out what was wrong.</p>

<p>Take a look at <code>yaml</code> definition of the deployment for the <code>php-services</code> service: (shortened)</p>



<pre><code>apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: php-services
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      name: php-services
  template:
    metadata:
      labels:
        name: php-services
    spec:
      containers:
        - name: php-services
          image: IMAGE_TAG
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /health
              port: 80
              scheme: HTTP
            initialDelaySeconds: 60
            periodSeconds: 60
            successThreshold: 1
            timeoutSeconds: 10
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /health
              port: 80
              scheme: HTTP
            initialDelaySeconds: 60
            periodSeconds: 60
            successThreshold: 1
            timeoutSeconds: 10
          ports:
          - containerPort: 80
</code></pre>

<p>The Apache server inside the image was configured in a way that it redirected from paths without trailing slash to paths with it. So when you requested <code>/health</code> you actually got HTTP status 301 pointing to <code>/health/</code> which then responded with 200.</p>

<p>In the scope of Kubernetes health checks this is OK as "<a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/" rel="noreferrer">Any code greater than or equal to 200 and less than 400 indicates success.</a>"</p>

<p>However, the problem lied in the GKE Loadbalancer. It also has it's own GKE healthchecks derived from the checks in Deployment definition. The important difference is that <a href="https://cloud.google.com/load-balancing/docs/health-check-concepts" rel="noreferrer">it only accepts HTTP status 200</a>. And if the loadbalancer doesn't find a backend service healthy it won't pass any foreign traffic to it.</p>

<p>Therefore we had two options to fix this:</p>

<ul>
<li>Make the server inside the container respond with HTTPS status 200 to both
<code>/health</code> and <code>/health/</code> (or more precisely just to <code>/health</code>)</li>
<li>or change the <code>readinessProbe</code> and <code>livenessProbe</code> path definition to <code>/health/</code>.</li>
</ul>

<p>We choose the later and it fixed the problem.</p>
