<p>I am actually trying to run 3 containers in my gke cluster. I have them exposed via a network load balancer and over that, I am using ingress so I can reach my services from different domains with SSL certs on them.</p>
<p>Here is the complete manifest</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app:web
    spec:
      containers:
      - name: web
        image: us-east4-docker.pkg.dev/web:e856485      # docker image
        ports:
        - containerPort: 3000
        env:
        - name: NODE_ENV
          value: production
---
# DEPLOYMENT MANIFEST #
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cms
spec:
  replicas: 3
  selector:
    matchLabels:
      app: cms
  template:
    metadata:
      labels:
        app: cms
    spec:
      containers:
      - name: cms
        image: us-east4-docker.pkg.dev/cms:4e1fe2f      # docker image
        ports:
        - containerPort: 8055
        env:
        - name  : DB
          value : &quot;postgres&quot;

        - name  : DB_HOST
          value : 10.142.0.3

        - name  : DB_PORT
          value : &quot;5432&quot;
---
# DEPLOYMENT MANIFEST #
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
      - name: api
        image: us-east4-docker.pkg.dev/api:4e1fe2f      # docker image
        ports:
        - containerPort: 8080
        env:
        - name  : HOST
          value : &quot;0.0.0.0&quot;

        - name  : PORT
          value : &quot;8080&quot;
     
        - name  : NODE_ENV
          value : production
---
# SERVICE MANIFEST #
apiVersion: v1
kind: Service
metadata:
  name: web-lb
  annotations:
    cloud.google.com/neg: '{&quot;ingress&quot;: true}'
  labels:
    app: web
spec:
  ports:
  - port: 3000
    protocol: TCP
    targetPort: 3000
  selector:
    app: web
  type: NodePort
---
# SERVICE MANIFEST #
apiVersion: v1
kind: Service
metadata:
  name: cms-lb
  annotations:
    cloud.google.com/neg: '{&quot;ingress&quot;: true}'
  labels:
    app: cms
spec:
  ports:
  - port: 8055
    protocol: TCP
    targetPort: 8055
  selector:
    app: cms
  type: NodePort
---
# SERVICE MANIFEST #
apiVersion: v1
kind: Service
metadata:
  name: api-lb
  annotations:
    cloud.google.com/neg: '{&quot;ingress&quot;: true}'
  labels:
    app: api
spec:
  ports:
  - port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    app: api
  type: NodePort
---
apiVersion: v1
data:
  tls.crt: abc
  tls.key: abc
kind: Secret
metadata:
  name: web-cert
type: kubernetes.io/tls
---
apiVersion: v1
data:
  tls.crt: abc
  tls.key: abc
kind: Secret
metadata:
  name: cms-cert
type: kubernetes.io/tls
---
apiVersion: v1
data:
  tls.crt: abc
  tls.key: abc
kind: Secret
metadata:
  name: api-cert
type: kubernetes.io/tls
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress
  annotations:
    # If the class annotation is not specified it defaults to &quot;gce&quot;.
    kubernetes.io/ingress.class: &quot;gce&quot;
spec:
  tls:
  - secretName: api-cert
  - secretName: cms-cert
  - secretName: web-cert
  rules:
  - host: web-gke.dev
    http:
      paths:
      - pathType: ImplementationSpecific
        backend:
          service:
            name: web-lb
            port:
              number: 3000
  - host: cms-gke.dev
    http:
      paths:
      - pathType: ImplementationSpecific
        backend:
          service:
            name: cms-lb
            port:
              number: 8055
  - host: api-gke.dev
    http:
      paths:
      - pathType: ImplementationSpecific
        backend:
          service:
            name: api-lb
            port:
              number: 8080
</code></pre>
<p>The containers are accessible through the load balancer(network), but from ingress(L7 lb) the health check is failing<a href="https://i.stack.imgur.com/DTLo9.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/DTLo9.png" alt="Ingress showing unhealthy services" /></a>.</p>
<p>I tried editing the health checks manually from HTTP:80 to TCP:8080/8055/3000 for 3 services and it works.<a href="https://i.stack.imgur.com/ujM6i.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/ujM6i.png" alt="ALB failing health checks" /></a></p>
<p>But eventually, ingress reverts it back to HTTP health check and it fails again. I also tried using NodePort instead of load balancer as service type but no luck.
Any help?</p>

Sohaib Mustafa
<p>The first thing I would like to mention is that you need to recheck your implementation because from what I see, you are creating an <code>Ingress</code> which will create a <code>LoadBanacer</code>, and this <code>Ingress</code> is using three services of type <code>LoadBalancer</code> in which each one of them will also create its <code>LoadBalancer</code> (I'm assuming the default behaviour, unless you applied the famous workaround of deleting the <code>service</code>'s <code>LoadBalancer</code> manually after it is created).</p>
<p>And I don't think this is correct unless you need that design for some reason. So, my suggestion is that you might want to change your services types to <code>NodePort</code>.</p>
<hr />
<p>As for answering your question, what you are missing is:</p>
<p>You need to implement a <code>BackendConfig</code> with custom <code>HealthCheck</code> configurations.</p>
<p>1- Create the <code>Backendconfig</code>:</p>
<pre><code>apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: api-lb-backendconfig
spec:
  healthCheck:
    checkIntervalSec: INTERVAL
    timeoutSec: TIMEOUT
    healthyThreshold: HEALTH_THRESHOLD
    unhealthyThreshold: UNHEALTHY_THRESHOLD
    type: PROTOCOL
    requestPath: PATH
    port: PORT
</code></pre>
<p>2- Use this config in your service/s</p>
<pre><code>apiVersion: v1
kind: Service
metadata:
  annotations:
    cloud.google.com/backend-config: '{&quot;ports&quot;: {
    &quot;PORT_NAME_1&quot;:&quot;api-lb-backendconfig&quot;
    }}'
spec:
  ports:
  - name: PORT_NAME_1
    port: PORT_NUMBER_1
    protocol: TCP
    targetPort: TARGET_PORT
</code></pre>
<p>Once you apply such configurations, your <code>Ingress</code>'s <code>LoadBalanacer</code> will be created with the <code>BackendConfig</code> <em>&quot;api-lb-backendconfig&quot;</em></p>
<hr />
<p>Consider <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features" rel="nofollow noreferrer">this</a> documentation page as your reference.</p>
