<p>I am deploying an application on GKE cluster and try to deploy a load balancer to make clients able to call this application.</p>
<p>My application spec is:</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  namespace: default
spec:
  replicas: 1

  selector:
    matchLabels:
      name: api
  template:
    metadata:
      labels:
        name: api
    spec:
      serviceAccountName: docker-sa
      containers:
        - name: api
          image: zhaoyi0113/rancher-go-api
          ports:
            - containerPort: 8080

apiVersion: v1
kind: Service
metadata:
  name: api
  annotations:
    cloud.google.com/neg: '{&quot;ingress&quot;: true}'
spec:
  selector:
    name: api
  ports:
    - port: 80
      targetPort: 8080
      protocol: TCP
  type: NodePort

</code></pre>
<p>It listens on the port 8080 and a service open port 80 and use the targetPort 8080 to connect to the application.</p>
<p>And I have a ingress spec:</p>
<pre><code>apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: sidecar
  namespace: default
spec:
  defaultBackend:
    service:
      name: api
      port:
        number: 80
</code></pre>
<p>After deploy, I am able to see the <code>ip address</code> from <code>kubectl get ingress</code>. But when I send a request to the ip, I got 502 error.</p>
<pre><code>$ kubectl get ingress
NAME      CLASS    HOSTS   ADDRESS           PORTS   AGE
sidecar   &lt;none&gt;   *       107.178.245.193   80      28m

$ kubectl describe ingress sidecar
Name:             sidecar
Labels:           &lt;none&gt;
Namespace:        default
Address:          107.178.245.193
Default backend:  api:80 (10.0.1.14:8080)
Rules:
  Host        Path  Backends
  ----        ----  --------
  *           *     api:80 (10.0.1.14:8080)
Annotations:  ingress.kubernetes.io/backends: {&quot;k8s1-5ae02eec-default-api-80-28d7bbec&quot;:&quot;Unknown&quot;}
              ingress.kubernetes.io/forwarding-rule: k8s2-fr-krllp0c9-default-sidecar-9a9n4r5m
              ingress.kubernetes.io/target-proxy: k8s2-tp-krllp0c9-default-sidecar-9a9n4r5m
              ingress.kubernetes.io/url-map: k8s2-um-krllp0c9-default-sidecar-9a9n4r5m
Events:
  Type    Reason     Age                  From                     Message
  ----    ------     ----                 ----                     -------
  Normal  Sync       29m                  loadbalancer-controller  UrlMap &quot;k8s2-um-krllp0c9-default-sidecar-9a9n4r5m&quot; created
  Normal  Sync       28m                  loadbalancer-controller  TargetProxy &quot;k8s2-tp-krllp0c9-default-sidecar-9a9n4r5m&quot; created
  Normal  Sync       28m                  loadbalancer-controller  ForwardingRule &quot;k8s2-fr-krllp0c9-default-sidecar-9a9n4r5m&quot; created
  Normal  IPChanged  28m                  loadbalancer-controller  IP is now 107.178.245.193
  Normal  Sync       3m51s (x7 over 29m)  loadbalancer-controller  Scheduled for sync

</code></pre>
<p>Below is the curl error response:</p>
<pre><code>$ curl -i http://107.178.245.193/health
HTTP/1.1 502 Bad Gateway
Content-Type: text/html; charset=UTF-8
Referrer-Policy: no-referrer
Content-Length: 332
Date: Tue, 16 Aug 2022 10:40:31 GMT


&lt;html&gt;&lt;head&gt;
&lt;meta http-equiv=&quot;content-type&quot; content=&quot;text/html;charset=utf-8&quot;&gt;
&lt;title&gt;502 Server Error&lt;/title&gt;
&lt;/head&gt;
&lt;body text=#000000 bgcolor=#ffffff&gt;
&lt;h1&gt;Error: Server Error&lt;/h1&gt;
&lt;h2&gt;The server encountered a temporary error and could not complete your request.&lt;p&gt;Please try again in 30 seconds.&lt;/h2&gt;
&lt;h2&gt;&lt;/h2&gt;
&lt;/body&gt;&lt;/html&gt;
</code></pre>
<p>When I describe the service <code>api</code>, I got below error:</p>
<pre><code>$ kubectl describe service api
Name:                     api
Namespace:                default
Labels:                   &lt;none&gt;
Annotations:              cloud.google.com/neg: {&quot;ingress&quot;: true}
                          cloud.google.com/neg-status: {&quot;network_endpoint_groups&quot;:{&quot;80&quot;:&quot;k8s1-29362abf-default-api-80-f2f1248a&quot;},&quot;zones&quot;:[&quot;australia-southeast2-a&quot;]}
                          field.cattle.io/publicEndpoints: [{&quot;port&quot;:30084,&quot;protocol&quot;:&quot;TCP&quot;,&quot;serviceName&quot;:&quot;default:api&quot;,&quot;allNodes&quot;:true}]
Selector:                 name=api
Type:                     NodePort
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.3.253.54
IPs:                      10.3.253.54
Port:                     &lt;unset&gt;  80/TCP
TargetPort:               8080/TCP
NodePort:                 &lt;unset&gt;  30084/TCP
Endpoints:                10.0.1.17:8080
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type     Reason        Age   From            Message
  ----     ------        ----  ----            -------
  Warning  AttachFailed  7s    neg-controller  Failed to Attach 2 network endpoint(s) (NEG &quot;k8s1-29362abf-default-api-80-f2f1248a&quot; in zone &quot;australia-southeast2-a&quot;): googleapi: Error 400: Invalid value for field 'resource.ipAddress': '10.0.1.18'. Specified IP address 10.0.1.18 doesn't belong to the (sub)network default or to the instance gke-gcp-cqrs-gcp-cqrs-node-pool-6b30ca5c-41q8., invalid
  Warning  RetryFailed   7s    neg-controller  Failed to retry NEG sync for &quot;default/api-k8s1-29362abf-default-api-80-f2f1248a--/80-8080-GCE_VM_IP_PORT-L7&quot;: maximum retry exceeded
</code></pre>
<p>Does anyone know what could be the root course?</p>

Joey Yi Zhao
<p>I created a new GKE cluster and tried setting up the same resources you are configuring. However, I used the following image for the container <code>gcr.io/google-samples/hello-app:1.0</code>. Everything else remains the same - leaving the <code>gcp-setup.yaml</code> file I used below for reference.</p>
<p><strong>gcp-setup.yaml</strong></p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  namespace: default
spec:
  replicas: 1

  selector:
    matchLabels:
      name: api
  template:
    metadata:
      labels:
        name: api
    spec:
      containers:
        - name: api
          image: gcr.io/google-samples/hello-app:1.0
          ports:
            - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: api
  annotations:
    cloud.google.com/neg: '{&quot;ingress&quot;: true}'
spec:
  selector:
    name: api
  ports:
    - port: 80
      targetPort: 8080
      protocol: TCP
  type: NodePort
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: sidecar
  namespace: default
spec:
  defaultBackend:
    service:
      name: api
      port:
        number: 80
</code></pre>
<p>There is also a small thing I had to change in your configuration, which is the <code>annotation</code> block - when I first tried to apply your configuration, I got the below error. Hence, I had to adjust the <code>annotation</code> entry to be <code>annotations</code>.</p>
<pre><code>&gt; kubectl apply -f gcp-setup.yaml
deployment.apps/api created
error: error validating &quot;gcp-setup.yaml&quot;: error validating data: ValidationError(Service.metadata): unknown field &quot;annotation&quot; in io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta; if you choose to ignore these errors, turn validation off with --validate=false
</code></pre>
<p>Afterwards, I was able to successfully provision all of the resources, and your configuration worked perfectly fine. It took around 3 minutes I believe for the Ingress resource to get an IP address assigned (masked as <code>XX.XXX.XXX.XX</code> below).</p>
<pre><code>&gt; kubectl get pods
NAME                   READY   STATUS    RESTARTS   AGE
api-7d6fdd9845-8dwqc   1/1     Running   0          7m13s

&gt; kubectl get services
NAME         TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)        AGE
api          NodePort    10.36.4.150   &lt;none&gt;        80:30142/TCP   7m1s
kubernetes   ClusterIP   10.36.0.1     &lt;none&gt;        443/TCP        12m

&gt; kubectl get ingress
NAME      CLASS    HOSTS   ADDRESS         PORTS   AGE
sidecar   &lt;none&gt;   *       XX.XXX.XXX.XX   80      7m18s

&gt; kubectl describe ingress
Name:             sidecar
Namespace:        default
Address:          XX.XXX.XXX.XX
Default backend:  api:80 (10.32.0.10:8080)
Rules:
  Host        Path  Backends
  ----        ----  --------
  *           *     api:80 (10.32.0.10:8080)
Annotations:  ingress.kubernetes.io/backends: {&quot;k8s1-05f3ce8b-default-api-80-82dd4d72&quot;:&quot;HEALTHY&quot;}
              ingress.kubernetes.io/forwarding-rule: k8s2-fr-9k4w4ytx-default-sidecar-9m5g4dex
              ingress.kubernetes.io/target-proxy: k8s2-tp-9k4w4ytx-default-sidecar-9m5g4dex
              ingress.kubernetes.io/url-map: k8s2-um-9k4w4ytx-default-sidecar-9m5g4dex
Events:
  Type    Reason     Age                 From                     Message
  ----    ------     ----                ----                     -------
  Normal  Sync       7m13s               loadbalancer-controller  UrlMap &quot;k8s2-um-9k4w4ytx-default-sidecar-9m5g4dex&quot; created
  Normal  Sync       7m10s               loadbalancer-controller  TargetProxy &quot;k8s2-tp-9k4w4ytx-default-sidecar-9m5g4dex&quot; created
  Normal  Sync       6m59s               loadbalancer-controller  ForwardingRule &quot;k8s2-fr-9k4w4ytx-default-sidecar-9m5g4dex&quot; created
  Normal  IPChanged  6m59s               loadbalancer-controller  IP is now XX.XXX.XXX.XX
  Normal  Sync       28s (x6 over 8m3s)  loadbalancer-controller  Scheduled for sync
</code></pre>
<p>After the Ingress resource became healthy, I was able to navigate in my browser to the assigned IP address <code>XX.XXX.XXX.XX</code> and got a successful response back from the workload I deployed (<code>gcr.io/google-samples/hello-app:1.0</code>).</p>
<p><strong>Browser Output</strong></p>
<pre><code>Hello, world!
Version: 1.0.0
Hostname: api-7d6fdd9845-8dwqc
</code></pre>
<p>As a conclusion, make sure to update your Service definition from <code>metadata.annotation</code> to <code>metadata.annotations</code>. It was the only change I had to do to make your configuration work. Furthermore, I recommend turning resource definition validation on to make sure that you catch such errors when defining new resources.</p>
<p>If the error still persists, I would recommend running <code>kubectl describe ingress sidecar</code> and analyze the output, assuming it is related to the Ingress resource definition.</p>
<p><strong>EDIT1</strong></p>
<p>To make sure that this is not a zone-related issue, I provisioned a VPC-native, Public cluster in the same zone that you are using (<code>australia-southeast2-a</code>). I then applied the same configuration, and it was successful, thus ruling out the zone-related topic.</p>
<p>Based on the additional information you included in the post, my best guess for some potential root causes for the Service error you're getting when running <code>kubectl describe service</code> would be:</p>
<ul>
<li>Your GKE cluster is not <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/alias-ips" rel="nofollow noreferrer">VPC-native</a> - I see this is a core requirement to be able to leverage NEG</li>
<li>Your GKE cluster has been provisioned as a Private cluster, and as a consequence, NEG tries to assign an IP address from the available Private subnet ranges. This would explain the <code>10.0.1.18</code> IP address that NEG tries to assign to the resource definition</li>
</ul>
