<p>I am trying to configure an nginx ingress for a GKE cluster and define a path on a configured subdomain. It seems that even if I am able to successfully ping the host, and the domain binding is done correctly, I keep getting a 404 back whenever I try to access the configured path.</p>
<p>My goal is to be able to have a single static IP configured for my ingress controller and expose multiple services on different paths.</p>
<p>Below you can find my deployment files - one more thing that I would add is that I am using Terraform to automate the configuration and deployment of GCP and Kubernetes resources.</p>
<p>After the GKE cluster is successfully provisioned, I first deploy the official nginx-ingress controller from <a href="https://docs.nginx.com/nginx-ingress-controller/installation/installation-with-helm/" rel="nofollow noreferrer">here</a> - below my Terraform script that configures and deploys the controller with a custom static IP that I provisioned on GCP.</p>
<pre><code>resource &quot;helm_release&quot; &quot;nginx&quot; {
  name  = &quot;nginx&quot;
  chart = &quot;nginx-stable/nginx-ingress&quot;
  timeout = 900

  set {
    name = &quot;controller.stats.enabled&quot;
    value = true
  }

  set {
    name  = &quot;controller.service.type&quot;
    value = &quot;LoadBalancer&quot;
  }

  set {
    name  = &quot;controller.service.loadBalancerIP&quot;
    value = &quot;&lt;MY_STATIC_IP_ADDRESS&gt;&quot;
  }
}
</code></pre>
<p>Below my ingress configuration that I also deploy via Terraform:</p>
<pre><code>resource &quot;kubernetes_ingress&quot; &quot;ingress&quot; {
  wait_for_load_balancer = true

  metadata {
    name = &quot;app-ingress&quot;

    annotations = {
        &quot;kubernetes.io/ingress.class&quot;: &quot;nginx&quot;
        &quot;nginx.ingress.kubernetes.io/rewrite-target&quot;: &quot;/&quot;
        &quot;kubernetes.io/ingress.global-static-ip-name&quot;: &lt;MY_STATIC_IP_ADDRESS&gt;
    }
  }

  spec {
    rule {
      host = custom.my_domain.com
      http {
        path {
          backend {
            service_name = &quot;app-service&quot;
            service_port = 5000
          }

          path = &quot;/app&quot;
        }
      }
    }
  }
}
</code></pre>
<p>And the resulting ingress configuration as taken from GCP:</p>
<pre><code>apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    kubernetes.io/ingress.global-static-ip-name: static-ip-name
    nginx.ingress.kubernetes.io/rewrite-target: /
  creationTimestamp: &quot;2021-04-14T20:28:41Z&quot;
  generation: 7
  name: app-ingress
  namespace: default
  resourceVersion: HIDDEN
  selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/app-ingress
  uid: HIDDEN
spec:
  rules:
  - host: custom.my_domain.com
    http:
      paths:
      - backend:
          serviceName: app-service
          servicePort: 5000
        path: /app
status:
  loadBalancer:
    ingress:
    - ip: &lt;MY_STATIC_IP_ADDRESS&gt;
</code></pre>
<p>And the output for the <code>kubectl describe ingress app-ingress</code> command:</p>
<pre><code>Name:             app-ingress
Namespace:        default
Address:          &lt;MY_STATIC_IP_ADDRESS&gt;
Default backend:  default-http-backend:80 (192.168.10.8:8080)
Rules:
  Host                  Path  Backends
  ----                  ----  --------
  custom.my_domain.com
                        /app   app-service:5000 (192.168.10.11:5000)
Annotations:            kubernetes.io/ingress.class: nginx
                        kubernetes.io/ingress.global-static-ip-name: static-ip-name
                        nginx.ingress.kubernetes.io/rewrite-target: /
Events:
  Type    Reason          Age                From                      Message
  ----    ------          ----               ----                      -------
  Normal  AddedOrUpdated  16m (x6 over 32m)  nginx-ingress-controller  Configuration for default/app-ingress was added or updated
</code></pre>
<p>I deployed the application that I am trying to expose by using the following configuration files:</p>
<p><strong>pvc.yaml</strong></p>
<pre><code>apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: app-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: default
</code></pre>
<p><strong>service.yaml</strong></p>
<pre><code>apiVersion: v1
kind: Service
metadata:
  name: app-service
spec:
  type: NodePort
  ports:
    - port: 5000
      targetPort: 5000
      protocol: TCP
      name: http
</code></pre>
<p><strong>deployment.yaml</strong></p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-deployment
spec:
  replicas: 1
  template:
    spec:
      restartPolicy: Always
      volumes:
        - name: app-pvc
          persistentVolumeClaim:
            claimName: app-pvc
      containers:
      - name: app-container
        image: &quot;eu.gcr.io/&lt;PROJECT_ID&gt;/&lt;IMAGE_NAME&gt;:VERSION_TAG&quot;
        imagePullPolicy: IfNotPresent
        ports:
          - containerPort: 5000
        livenessProbe:
          tcpSocket:
            port: 5000
          initialDelaySeconds: 10
          periodSeconds: 10
        readinessProbe:
          tcpSocket:
            port: 5000
          initialDelaySeconds: 15
          periodSeconds: 20
        volumeMounts:
          - mountPath: &quot;/data&quot;
            name: app-pvc
</code></pre>
<p>Everything gets deployed successfully, as I am able to directly connect to the application locally via the configured service by running the following command:</p>
<pre><code>kubectl port-forward service/app-service 5000:5000
</code></pre>
<p>This allows me to access the application in my browser and everything works as intended.</p>
<p>To make sure that <code>&lt;MY_STATIC_IP_ADDRESS&gt;</code> that I configured is properly bound to <code>custom.my_domain.com</code>, I tried to ping the host and I do get the right response back:</p>
<pre><code>ping custom.my_domain.com

Pinging custom.my_domain.com [&lt;MY_STATIC_IP_ADDRESS&gt;] with 32 bytes of data:
Reply from &lt;MY_STATIC_IP_ADDRESS&gt;: bytes=32 time=36ms TTL=113
Reply from &lt;MY_STATIC_IP_ADDRESS&gt;: bytes=32 time=37ms TTL=113
Reply from &lt;MY_STATIC_IP_ADDRESS&gt;: bytes=32 time=36ms TTL=113
Reply from &lt;MY_STATIC_IP_ADDRESS&gt;: bytes=32 time=45ms TTL=113

Ping statistics for &lt;MY_STATIC_IP_ADDRESS&gt;:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 36ms, Maximum = 45ms, Average = 38ms
</code></pre>
<p>Even if everything appears to be working as intended, whenever I try to navigate to <code>custom.my_domain.com/app</code> in my browser, I keep getting the following response in my browser, even after waiting for more than 30m to make sure that the ingress configuration has been properly registered on GCP:
<a href="https://i.stack.imgur.com/ghQOn.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/ghQOn.png" alt="enter image description here" /></a></p>
<p>And this is the entry that shows up in the logs of my nginx-controller pod:</p>
<pre><code>&lt;HIDDEN_LOCAL_IP&gt; - - [14/Apr/2021:21:18:10 +0000] &quot;GET /app/ HTTP/1.1&quot; 404 232 &quot;-&quot; &quot;Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0&quot; &quot;-&quot;
</code></pre>
<p><strong>UPDATE #1</strong></p>
<p>It appears that if I update my ingress to directly expose the targeted service on the <code>/</code> path, it works as intended. Below the updated configuration. Still, it appears that if I try to set any other path, it does not work.</p>
<pre><code>apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    kubernetes.io/ingress.global-static-ip-name: static-ip-name
    nginx.ingress.kubernetes.io/rewrite-target: /
  creationTimestamp: &quot;2021-04-14T20:28:41Z&quot;
  generation: 7
  name: app-ingress
  namespace: default
  resourceVersion: HIDDEN
  selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/app-ingress
  uid: HIDDEN
spec:
  rules:
  - host: custom.my_domain.com
    http:
      paths:
      - backend:
          serviceName: app-service
          servicePort: 5000
        path: /
status:
  loadBalancer:
    ingress:
    - ip: &lt;MY_STATIC_IP_ADDRESS&gt;

</code></pre>
<p><strong>Update #2</strong></p>
<p>After going through the materials shared by @jccampanero in the comments section, I was able to get a working configuration.</p>
<p>Instead of using <a href="https://docs.nginx.com/nginx-ingress-controller/installation/installation-with-helm/" rel="nofollow noreferrer">nginx-stable</a> which is referenced on the official nginx website, I used the one <a href="https://kubernetes.github.io/ingress-nginx" rel="nofollow noreferrer">here</a> and updated my Terraform script accordingly to use this one with the exact same configuration I had.</p>
<p>Afterwards, I had to update my ingress by following the documentation <a href="https://kubernetes.github.io/ingress-nginx/examples/rewrite/#rewrite-target" rel="nofollow noreferrer">here</a> - below the updated configuration:</p>
<pre><code>apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    kubernetes.io/ingress.global-static-ip-name: static-ip-name
    nginx.ingress.kubernetes.io/rewrite-target: /$2
  creationTimestamp: &quot;2021-04-14T20:28:41Z&quot;
  generation: 7
  name: app-ingress
  namespace: default
  resourceVersion: HIDDEN
  selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/app-ingress
  uid: HIDDEN
spec:
  rules:
  - host: custom.my_domain.com
    http:
      paths:
      - backend:
          serviceName: app-service
          servicePort: 5000
        path: /app(/|$)(.*)
status:
  loadBalancer:
    ingress:
    - ip: &lt;MY_STATIC_IP_ADDRESS&gt;

</code></pre>

<p>As indicated in the question comments and in the question itself, very well documented by @vladzam, two are the reasons of the problem.</p>
<p>On one hand, the nginx ingress controller available through the Helm <code>stable</code> channel seems to be <a href="https://github.com/helm/charts/tree/master/stable/nginx-ingress" rel="nofollow noreferrer">deprecated</a> in favor of the new <code>ingress-nginx</code> controller - please, see <a href="https://github.com/kubernetes/ingress-nginx/tree/master/charts/ingress-nginx%60" rel="nofollow noreferrer">the Github repo</a> and the <a href="https://kubernetes.github.io/ingress-nginx/" rel="nofollow noreferrer">official documentation</a>.</p>
<p>On the other, it seems to be a problem related to the definition of the <code>Rewrite target</code> annotation. According to the <a href="https://kubernetes.github.io/ingress-nginx/examples/rewrite/#rewrite-target" rel="nofollow noreferrer">docs</a>:</p>
<blockquote>
<p>Starting in Version 0.22.0, ingress definitions using the annotation <code>nginx.ingress.kubernetes.io/rewrite-target</code> are not backwards compatible with previous versions. In Version 0.22.0 and beyond, any substrings within the request URI that need to be passed to the rewritten path must explicitly be defined in a capture group.</p>
</blockquote>
<p>As a consequence, it is necessary to modify the definition of the ingress resource to take into account this change. For instance:</p>
<pre class="lang-yaml prettyprint-override"><code>$ echo '
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
  name: rewrite
  namespace: default
spec:
  rules:
  - host: rewrite.bar.com
    http:
      paths:
      - backend:
          serviceName: http-svc
          servicePort: 80
        path: /something(/|$)(.*)
' | kubectl create -f -
</code></pre>
<p>The question itself provides the exact ingress resource definition.</p>
