<p>TL;DR
In a GKE private cluster, I'm unable to expose service with internal/private IP.</p>

<p>We have our deployment consisting of around 20 microservices and 4 monoliths, currently running entirely on VMs on GoogleCloud. I'm trying to move this infrastructure to GKE. The first step of the project is to build a private GKE cluster (i.e without any public IP) as a replacement of our staging. As this is staging, I need to expose all the microservice endpoints along with the monolith endpoints internally for debugging purpose (means, only to those connected to the VPC) and that is where I'm stuck. I tried 2 approaches:</p>

<ol>
<li>Put an internal load balancer (ILB) in front of each service and monolith. Example:</li>
</ol>

<pre><code>apiVersion: v1
kind: Service
metadata:
  name: session
  annotations:
    cloud.google.com/load-balancer-type: "Internal"
  labels:
    app: session
    type: ms
spec:
  type: LoadBalancer
  selector:
    app: session
  ports:
  - name: grpc
    port: 80
    targetPort: 80
    protocol: TCP
</code></pre>

<p><a href="https://i.stack.imgur.com/R4Ndo.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/R4Ndo.png" alt="enter image description here"></a></p>

<p>This works, though with severe limitation. ILB creates a forwarding rule, and <strong>GCP has a <a href="https://cloud.google.com/vpc/docs/quota#per_network" rel="nofollow noreferrer">limitation</a> of 75 forwarding rule per network</strong>. This means we can not build more than 3 clusters in a network. Not acceptable to us.</p>

<ol start="2">
<li>a. I tried placing an ingress controller in front of all the services, which always exposes the entire cluster with a public IP - an absolute no-no.</li>
</ol>

<pre><code>apiVersion: extensions/v1beta1
kind: Ingress
hostNetwork: true
metadata:
  name: ingress-ms-lb
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: gs03
    http:
      paths:
      - path: /autodelivery/*
        backend:
          serviceName: autodelivery
          servicePort: 80
      - path: /session/*
        backend:
          serviceName: session
          servicePort: 80
</code></pre>

<p>b. I tried using a nginx ingress controller which ends up not having an ip at all.</p>

<pre><code>
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-ms-lb
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    # cloud.google.com/load-balancer-type: "Internal"
    nginx.ingress.kubernetes.io/ingress.class: nginx
    kubernetes.io/ingress.class: "nginx"
    # nginx.ingress.kubernetes.io/whitelist-source-range: 10.100.0.0/16, 10.110.0.0/16
spec:
  rules:
  - host: svclb
    http:
      paths:
      - path: /autodelivery/*
        backend:
          serviceName: autodelivery
          servicePort: 80
      - path: /session/*
        backend:
          serviceName: session
          servicePort: 80
</code></pre>

<p>The third option is to configure firewall rules, which will cut off any access to the public IPs. This was rejected internally, given the security concerns.</p>

<p>I'm stuck at this stage and need some pointers to move forward. Please help</p>

smaikap
<p>I could see from the screenshot that you attached that your GKE cluster is a private cluster.</p>

<p>As you would like to reach your services and applications inside the GKE Cluster from all the resources in the same VPC Network, I would like to suggest you to use NodePort [1].</p>

<p>[1] <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/service#service_of_type_nodeport" rel="nofollow noreferrer">https://cloud.google.com/kubernetes-engine/docs/concepts/service#service_of_type_nodeport</a></p>
