<p>We have an GKE ingress that is using the below frontend-service. The ingress terminates tls as well. We want to have http to https permanent redirects for any traffic that comes on http. </p>

<p>With the below configuration we have all working, and serving traffic on both http and https (without redirect).</p>

<p>The container used for the Deployment can be configured to rewrite http to https with --https-redirect flag. It also respect and trust the <strong>X-Forwarded-Proto</strong> header, and will consider it to be secure if the header value is set to <strong>https</strong>.</p>

<p>So the most reasonable configuration I can see for the readinessProbe would be the configuration below, but with the commented lines uncommented. However, as soon as we apply this version we never get into a healthy state, and instead the terminated domain configured with the Ingress returns with 502 and never recovers.</p>

<p>So what is wrong with the below approach?
I have tested using port-forwarding both the pod and the service, and they both return 301 if I do not provide the X-Forwarded-Proto header, and return 200 if I provide the X-Forwarded-Proto header with https value.</p>

<pre><code>apiVersion: v1
kind: Service
metadata:
  name: frontend-service
spec:
  type: NodePort
  ports:
    - port: 8080
  selector:
    app: frontend
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend
spec:
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
    spec:
      containers:
      - image: eu.gcr.io/someproject/frontend:master
        imagePullPolicy: Always
        # args:
        # - '--https-redirect'
        name: frontend
        resources:
          limits:
            memory: 1Gi
            cpu: '0.5'
        ports:
        - containerPort: 8080
          name: frontend
        readinessProbe:
          httpGet:
            path: /_readinessProbe
            port: 8080
            # httpHeaders:
            # - name: X-Forwarded-Proto
            #   value: https
</code></pre>

<p>The GCP Health Check is very picky about the HTTP response codes it gets back. It must be a 200, and not a redirect. If in the configuration you have posted, the NLB gets a 301/302 response from your server. it will then mark your backend as unhealthy as this is not a 200 response. If the health check is sending HTTP without the X-Forwarded-Proto header, this is likely.</p>

<p>You can check this by inspecting the kubectl logs of your deployment's pods.</p>

<p>My previous answer may be useful if you move to an HTTPS health check, in an attempt to remedy this.</p>

<hr>

<p>From <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/ingress" rel="nofollow noreferrer">GKE documentation</a>:
You will need to put an annotation on your Service definition that tells GKE to use HTTPS for the health check. Otherwise it will try sending HTTP and get confused.</p>

<pre><code>kind: Service
metadata:
  name: my-service-3
  annotations:
    cloud.google.com/app-protocols: '{"my-https-port":"HTTPS","my-http-port":"HTTP"}'
spec:
  type: NodePort
  selector:
    app: metrics
    department: sales
  ports:
  - name: my-https-port
    port: 443
    targetPort: 8443
  - name: my-http-port
    port: 80
    targetPort: 50001
</code></pre>

<p>I haven't used the latest syntax, but this used to work for me.</p>

<p>However this was so clunky to use I ended up going over to Istio and getting that to do all the HTTPS termination. That's no small undertaking however, but it you're thinking of using cert-manager/Let's Encrypt it might be worth exploring.</p>
