<p>I have REST API Web service on Internal GKE cluster which I would like to expose with internal HTTP load balancing.</p>
<p>Let's call this service &quot;blue&quot; service:
I would like to expose it in following mapping:</p>
<pre><code>http://api.xxx.yyy.internal/blue/isalive -&gt; http://blue-service/isalive
http://api.xxx.yyy.internal/blue/v1/get -&gt; http://blue-service/v1/get
http://api.xxx.yyy.internal/blue/v1/create -&gt; http://blue-service/v1/create
http://api.xxx.yyy.internal/ -&gt; http://blue-service/ (expose Swagger)
</code></pre>
<p>I'm omitting deployment yaml, since it's less relevant to discussion.</p>
<p>But my service yaml looks like this:</p>
<pre><code>apiVersion: v1
kind: Service
metadata:
  name: blue-service
spec:
  type: NodePort
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: blue-service
</code></pre>
<p>My Ingress configuration is the following:</p>
<pre><code>apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: blue-ingress
  annotations:
    kubernetes.io/ingress.class: &quot;gce-internal&quot;
    nginx.ingress.kubernetes.io/ssl-redirect: &quot;false&quot;
    ingress.kubernetes.io/rewrite-target: /$2
spec:
  rules:
  - host: api.xxx.yyy.internal
    http:
      paths:
      - path: /blue/*
        backend:
          serviceName: blue-service
          servicePort: 80
</code></pre>
<p>However, I'm receiving 404 for all requests. <code>/blue/v1/get</code>, <code>/blue/v1/create</code> and <code>/blue/isalive</code> returns 404.</p>
<p>In my &quot;blue&quot; application I log all my notFound requests and I can clearly see that my URIs are not being rewritten, the requests hitting the application are <code>/blue/v1/get</code>, <code>/blue/v1/create</code> and <code>/blue/isalive</code>.</p>
<p>What am I missing in Ingress configuration? How can I fix those rewrites?</p>

<p>I solved the problem and writing it here to memo it and hopefully someone will find it as useful.</p>
<ul>
<li><p>First problem is that I have mixed annotations types. one of GKE ingress controller and second for Nginx Server controller. Currently GKE ingress controller doesn't support URL rewrite feature, so I need to use nginx ingress controller.</p>
</li>
<li><p>so I need to install Nginx based ingress controller. It cloud be done easily using Helm chart or or deployment yaml. However, by default this controller will expose ingress using external load balancer and this not what I want. So we need to modify deployment charts or YAML file of this controller.
I'm not using Helm, so I downoaded yaml itself using wget command.</p>
</li>
</ul>
<pre><code>
    wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.47.0/deploy/static/provider/cloud/deploy.yaml
</code></pre>
<p>Open it in editor and find the definition of Service names <code>ingress-nginx-controller</code> in namespace <code>ingress-nginx</code>. Add the following annotation.</p>
<pre><code>cloud.google.com/load-balancer-type: &quot;Internal&quot;  
</code></pre>
<p>After it I can run <code>kubectl apply -f deploy.yaml</code> command which will create Ingress controller for me. It will take a few minutes to provision it.</p>
<ul>
<li><p>In addition I need to open firewall rule which will allow master nodes access worker nodes on port <code>8443/tcp</code>.</p>
</li>
<li><p>And the last item is an ingress yaml itself which should look like this:</p>
 <pre><code>
 apiVersion: networking.k8s.io/v1beta1
 kind: Ingress
 metadata:
   annotations:
     nginx.ingress.kubernetes.io/rewrite-target: /$2
     kubernetes.io/ingress.class: "nginx"
   name: blue-ingress
   namespace: default
 spec:
   rules:
   - host: api.xxx.yyy.internal
     http:
       paths:
       - backend:
           serviceName: blue-service
           servicePort: 80
         path: /blue(/|$)(.*)</code></pre>
</li>
</ul>
