<p>I have a StatefulSet, Service with NEG, and Ingress set up on Google Cloud Kubernetes Engine cluster. </p>

<p>Every workload and network object is ready and healthy. Ingress is created and NEG status is updated for all the services. VPC-native (Alias-IP) and HTTP Load Balancer options are enabled for the cluster. </p>

<p>But when I try to access my application using a path specified in my Ingress I always get 502 (Bad Gateway) error.</p>

<p>Here is my configuration (names are redacted including image name):</p>

<pre><code>apiVersion: v1
kind: Service
metadata:
  annotations:
    cloud.google.com/neg: '{"ingress": true}'
  labels:
    app: myapp
  name: myapp
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: tcp
  selector:
    app: myapp
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app: myapp
  name: myapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myapp
  serviceName: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: myapp
        livenessProbe:
          httpGet:
            path: /
            port: tcp
            scheme: HTTP
          initialDelaySeconds: 60
        image: myapp:8bebbaf
        ports:
        - containerPort: 1880
          name: tcp
          protocol: TCP
        readinessProbe:
          failureThreshold: 1
          httpGet:
            path: /
            port: tcp
            scheme: HTTP
        volumeMounts:
        - mountPath: /data
          name: data
      securityContext:
        fsGroup: 1000
      terminationGracePeriodSeconds: 10
  volumeClaimTemplates:
  - metadata:
      labels:
        app: myapp
      name: data
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 1Gi
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: myapp-ingress
spec:
  rules:
  - http:
      paths:
      - path: /workflow
        backend:
          serviceName: myapp
          servicePort: 80

</code></pre>

<p>What's wrong with it and how can I fix it?</p>

sorjef
<p>After much digging and tests I finally found what's wrong. Also, it seems like GKE NEG Ingress is not very stable (indeed NEG is in beta) and does not always conform to Kubernetes specs. </p>

<p>There was <a href="https://github.com/kubernetes/ingress-gce/issues/832" rel="nofollow noreferrer">an issue with GKE Ingress</a> related to named ports in <code>targetPort</code> field. The fix is implemented and available from <strong>1.16.0-gke.20</strong> cluster version (<a href="https://github.com/kubernetes/ingress-gce/releases/tag/v1.8.0" rel="nofollow noreferrer">Release</a>), which as of today (February 2020) is available under Rapid Channel, but I have not tested the fix as I had other issues with an ingress on a version from this channel.</p>

<p>So basically there are 2 options if you experience the same issue:</p>

<ol>
<li><p>Specify exact port number and not port name in a <code>targetPort</code> field in your service. Here is a fixed service config file from my example:</p>

<pre><code>apiVersion: v1
kind: Service
metadata:
  annotations:
    cloud.google.com/neg: '{"ingress": true}'
  labels:
    app: myapp
  name: myapp
spec:
  ports:
  - port: 80
    protocol: TCP
    # !!!
    # targetPort: tcp
    targetPort: 1088
  selector:
    app: myapp
</code></pre></li>
<li><p>Upgrade GKE cluster to 1.16.0-gke.20+ version (haven't tested it myself).</p></li>
</ol>
