<p><em>I’ve already seen <a href="https://stackoverflow.com/q/44584270/1813169">this question</a>; AFAIK I’m doing everything in the answers there.</em></p>

<p>Using GKE, I’ve deployed a GCP HTTP(S) load balancer-based ingress for a kubernetes cluster containing two almost identical deployments: production and development instances of the same application.</p>

<p>I set up a dedicated port on each pod template to use for health checks by the load balancer so that they are not impacted by redirects from the root path on the primary HTTP port. However, the health checks are consistently failing.</p>

<p>From <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/ingress#health_checks" rel="nofollow noreferrer">these</a> <a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/#probe-v1-core" rel="nofollow noreferrer">docs</a> I added a <code>readinessProbe</code> parameter to my deployments, which the load balancer seems to be ignoring completely.</p>

<p>I’ve verified that the server on <code>:p-ready</code> (9292; the dedicated health check port) is running correctly using the following (in separate terminals):</p>

<pre class="lang-none prettyprint-override"><code>➜ kubectl port-forward deployment/d-an-server p-ready
➜ curl http://localhost:9292/ -D -
HTTP/1.1 200 OK
content-length: 0
date: Wed, 26 Feb 2020 01:21:55 GMT
</code></pre>

<p><strong>What have I missed?</strong></p>

<p>A couple notes on the below configs:</p>

<ul>
<li>The <code>${...}</code> variables below are filled by the build script as part of deployment.</li>
<li>The second service (<code>s-an-server-dev</code>) is almost an exact duplicate of the first (with it’s own deployment) just with <code>-dev</code> suffixes on the names and labels.</li>
</ul>

<h3>Deployment</h3>

<pre class="lang-yaml prettyprint-override"><code>apiVersion: "apps/v1"
kind: "Deployment"
metadata:
  name: "d-an-server"
  namespace: "default"
  labels:
    app: "a-an-server"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: "a-an-server"
  template:
    metadata:
      labels:
        app: "a-an-server"
    spec:
      containers:
        - name: "c-an-server-app"
          image: "gcr.io/${PROJECT_ID}/an-server-app:${SHORT_SHA}"
          ports:
            - name: "p-http"
              containerPort: 8080
            - name: "p-ready"
              containerPort: 9292
          readinessProbe:
            httpGet:
              path: "/"
              port: "p-ready"
            initialDelaySeconds: 30
</code></pre>

<h3>Service</h3>

<pre class="lang-yaml prettyprint-override"><code>apiVersion: "v1"
kind: "Service"
metadata:
  name: "s-an-server"
  namespace: "default"
spec:
  ports:
    - port: 8080
      targetPort: "p-http"
      protocol: "TCP"
      name: "sp-http"
  selector:
    app: "a-an-server"
  type: "NodePort"
</code></pre>

<h3>Ingress</h3>

<pre class="lang-yaml prettyprint-override"><code>apiVersion: "networking.k8s.io/v1beta1"
kind: "Ingress"
metadata:
  name: "primary-ingress"
  annotations:
    kubernetes.io/ingress.global-static-ip-name: "primary-static-ipv4"
    networking.gke.io/managed-certificates: "appname-production-cert,appname-development-cert"
spec:
  rules:
    - host: "appname.example.com"
      http:
        paths:
          - backend:
              serviceName: "s-an-server"
              servicePort: "sp-http"
    - host: "dev.appname.example.com"
      http:
        paths:
          - backend:
              serviceName: "s-an-server-dev"
              servicePort: "sp-http-dev"
</code></pre>

MTCoster
<p>I think what's happening here is GKE ingress is not at all informed of port 9292. You are referring <code>sp-http</code> in the ingress which refers to port 8080.</p>

<p>You need to make sure of below:</p>

<p>1.The service's targetPort field must point to the pod port's containerPort value or name.</p>

<p>2.The readiness probe must be exposed on the port matching the servicePort specified in the Ingress.</p>
