<p>I am using EKS cluster with kubernetes version 1.18 and nginx igress controller. I have created below delpoyment, service and ingress resource what I have noticed ingress resource is not working for paths with regex and it is always returning 404 from ingress controller itself without passing request to backend pod. I have verified using describe I am able to see proper backends.</p>
<p>Can you please help on this.</p>
<h2>deployment.yaml</h2>
<pre><code>apiVersion: apps/v1
   kind: Deployment
   metadata:
      name: &quot;wildfly&quot;
      namespace: &quot;sit-web-n&quot;
   spec:
      selector:
        matchLabels:
        app: &quot;wildfly&quot;
      replicas: 1
      template:
        metadata:
      labels:
        app: &quot;wildfly&quot;
     spec:
      containers:
      - image: &lt;account&gt;.dkr.ecr.eu-west-1.amazonaws.com/ecr-test:wildfly
        imagePullPolicy: Always
        name: &quot;wildfly&quot;
        env:
        - name: JAVA_OPTS
          value: &quot;-Xms128m -Xmx768m&quot;
        ports:
        - containerPort: 8080
</code></pre>
<h2>service.yaml</h2>
<pre><code>apiVersion: v1
kind: Service
metadata:
  name: &quot;wildfly&quot;
  namespace: &quot;sit-web-n&quot;
spec:
  ports:
    - port: 8080
      targetPort: 8080
      protocol: TCP
  type: NodePort
  selector:
    app: &quot;wildfly&quot;
</code></pre>
<h2>ingress.yaml</h2>
<pre><code>apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: wildfly
  namespace: &quot;sit-web-n&quot;
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: &quot;false&quot;
    nginx.ingress.kubernetes.io/force-ssl-redirect: &quot;false&quot;
    nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
  ingressClassName: nginx
  tls:
    - hosts:
      - wildfly.eks-test.co.uk
      secretName: tls-secret
   rules:
   - host: wildfly.eks-test.co.uk
    http:
      paths:
      - path: /customer-svc/?(.*)
        pathType: Prefix
        backend:
          serviceName: wildfly
          servicePort: 8080
</code></pre>
<h2>Service with backend</h2>
<pre><code>kubectl describe svc wildfly  -n sit-web-n
Name:                     wildfly
Namespace:                sit-web-n
Labels:                   &lt;none&gt;
Annotations:              Selector:  app=wildfly
Type:                     NodePort
IP:                       10.100.78.242
Port:                     &lt;unset&gt;  8080/TCP
TargetPort:               8080/TCP
NodePort:                 &lt;unset&gt;  31350/TCP
Endpoints:                192.168.0.89:8080
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   &lt;none&gt;
[test]#
</code></pre>
<h2>ingress</h2>
<pre><code> kubectl describe ingress wildfly  -n sit-web-n
   Name:             wildfly
   Namespace:        sit-web-n
   Address:
   Default backend:  default-http-backend:80 (&lt;error: endpoints &quot;default-http-backend&quot; not found&gt;)
    TLS:
     tls-secret terminates wildfly.eks-test.co.uk
    Rules:
     Host                    Path  Backends
      ----                    ----  --------
     wildfly.eks-test.co.uk
                            /customer-svc/?(.*)   wildfly:8080 (192.168.0.89:8080)
   Annotations:              nginx.ingress.kubernetes.io/force-ssl-redirect: false
                             nginx.ingress.kubernetes.io/rewrite-target: /$1
                             nginx.ingress.kubernetes.io/ssl-redirect: false
    Events:
     Type    Reason          Age   From                      Message
      ----    ------          ----  ----                      -------
      Normal  AddedOrUpdated  14m   nginx-ingress-controller  Configuration for sit-web-n/wildfly was added or updated
</code></pre>
<h2>curl output</h2>
<pre><code>curl -ik -H 'Host: wildfly.eks-test.co.uk' https://ip-192-168-0-101.eu-west-1.compute.internal/custom- 
   svc/health
    HTTP/1.1 404 Not Found
    Server: nginx
    Date: Tue, 17 Nov 2020 08:31:08 GMT
   Content-Type: text/html
   Content-Length: 146
   Connection: keep-alive
   Strict-Transport-Security: max-age=15724800; includeSubDomains; preload

   &lt;html&gt;
   &lt;head&gt;&lt;title&gt;404 Not Found&lt;/title&gt;&lt;/head&gt;
   &lt;body&gt;
   &lt;center&gt;&lt;h1&gt;404 Not Found&lt;/h1&gt;&lt;/center&gt;
   &lt;hr&gt;&lt;center&gt;nginx&lt;/center&gt;
   &lt;/body&gt;
   &lt;/html&gt;



nginx ingress controller logs -

2020/11/17 08:49:44 [error] 52#52: *17 open() &quot;/etc/nginx/html/custom-svc/health&quot; failed (2: No such file or directory), client: 172.31.0.183, server: wildfly.eks-test.co.uk, request: &quot;GET /custom-svc/health HTTP/1.1&quot;, host: &quot;wildfly.eks-test.co.uk&quot;
wildfly.eks-test.co.uk 172.31.0.183 - - [17/Nov/2020:08:49:44 +0000] GET /custom-svc/health HTTP/1.1 404 146 0.000 - curl/7.61.1 - - -
2020/11/17 08:49:48 [error] 52#52: *18 open() &quot;/etc/nginx/html/custom-svc/health&quot; failed (2: No such file or directory), client: 172.31.0.183, server: wildfly.eks-test.co.uk, request: &quot;GET /custom-svc/health HTTP/1.1&quot;, host: &quot;wildfly.eks-test.co.uk&quot;
wildfly.eks-test.co.uk 172.31.0.183 - - [17/Nov/2020:08:49:48 +0000] GET /custom-svc/health HTTP/1.1 404 146 0.000 - curl/7.61.1 - - -
   
</code></pre>

R D
<p>As stated in the comment:</p>
<p><strong>1</strong>. Inside <strong>ingress resource</strong> you have <strong>defined path</strong>: <strong>customer-svc</strong> while trying to reach <strong>custom-svc</strong></p>
<p><strong>2</strong>. <code>&quot;/etc/nginx/html/custom-svc/health&quot; failed (2: No such file or directory)&quot;</code>. Normally such errors inform us that there is no such path on the underlying backend (strange for nginx ingress controller).</p>
<hr />
<p>It's hard to figure all possibilities but try to follow this:</p>
<p>Make sure you have installed and configured proper nginx controller. Different controllers and versions have different configurations and options like:</p>
<ul>
<li><p><a href="https://kubernetes.github.io/ingress-nginx/deploy/#aws" rel="nofollow noreferrer">nginx ingress (kubernetes)</a></p>
</li>
<li><p><a href="https://github.com/nginxinc/kubernetes-ingress" rel="nofollow noreferrer">kubernetes-ingress (nginxinc)</a></p>
</li>
<li><p>Note about: <strong>Rewrite Target</strong>:</p>
</li>
</ul>
<blockquote>
<p>attention Starting in Version 0.22.0, ingress definitions using the annotation nginx.ingress.kubernetes.io/rewrite-target are not backwards compatible with previous versions. In Version 0.22.0 and beyond, any substrings within the request URI that need to be passed to the rewritten path must explicitly be defined in a <a href="https://github.com/kubernetes/ingress-nginx/blob/master/docs/examples/rewrite/README.md#rewrite-target" rel="nofollow noreferrer">capture group.</a></p>
</blockquote>
<p>In addition please take a look at:</p>
<ul>
<li><code>ingressClassName</code> <a href="https://v1-18.docs.kubernetes.io/docs/concepts/services-networking/ingress/#ingress-class" rel="nofollow noreferrer">configuration,</a></li>
<li><code>Default Ingress Class</code> <a href="https://v1-18.docs.kubernetes.io/docs/concepts/services-networking/ingress/#default-ingress-class" rel="nofollow noreferrer">configuration,</a></li>
</ul>
<blockquote>
<p>You can mark a particular IngressClass as default for your cluster. Setting the ingressclass.kubernetes.io/is-default-class annotation to true on an IngressClass resource will ensure that new Ingresses without an ingressClassName field specified will be assigned this default IngressClass.</p>
</blockquote>
<hr />
<p><strong>Update</strong>:</p>
<p>As stated above: &quot;<strong>Different controllers and versions have different configurations and options</strong>&quot;.
You are using <a href="https://github.com/nginxinc/kubernetes-ingress" rel="nofollow noreferrer">nginxinc kubernetes-ingress controller</a> which is different from the <strong>NGINX Ingress controller</strong> in <strong>kubernetes/ingress-nginx</strong> repo. Please take a look at this <a href="https://github.com/nginxinc/kubernetes-ingress/blob/master/docs/nginx-ingress-controllers.md" rel="nofollow noreferrer">doc</a> to find out about the key differences.</p>
<p>Those annotations:</p>
<pre><code>nginx.ingress.kubernetes.io/ssl-redirect: &quot;false&quot;
nginx.ingress.kubernetes.io/force-ssl-redirect: &quot;false&quot;
nginx.ingress.kubernetes.io/rewrite-target: /$1
</code></pre>
<p>were taken from <a href="https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#annotations" rel="nofollow noreferrer">nginx-ingress (kubernetes)</a> and thus will not work with <a href="https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations/" rel="nofollow noreferrer">nginxinc kubernetes-ingress controller</a>.</p>
<p>As an example for uri rewriting please use <a href="https://github.com/nginxinc/kubernetes-ingress/tree/v1.9.1/examples/rewrites" rel="nofollow noreferrer">nginx.org/rewrites</a> annotation:</p>
<pre><code>annotations:
  nginx.org/rewrites: &quot;serviceName=wildfly rewrite=/&quot;
</code></pre>
<p>In addition, for using <strong>nginxinc kubernetes-ingress controller</strong> with regex and advanced routing support please take a look at <a href="https://docs.nginx.com/nginx-ingress-controller/configuration/virtualserver-and-virtualserverroute-resources/#virtualserver-route" rel="nofollow noreferrer">VirtualServer.Route</a> custom resource definition (CRD) concept. This functionality requires to install/configure <a href="https://github.com/nginxinc/kubernetes-ingress/tree/master/deployments/helm-chart#installing-the-crds" rel="nofollow noreferrer">CRD support</a> during controller installation.</p>
<p><a href="https://github.com/nginxinc/kubernetes-ingress/tree/v1.9.1/examples-of-custom-resources/rewrites" rel="nofollow noreferrer">Here</a> you can find complete guide how it should be configured using VirtualServer CRD  for nginxinc kubernetes-ingress controller.</p>
<hr />
<p>Additional information:</p>
<ul>
<li><a href="https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/#additional-controllers" rel="nofollow noreferrer">controllers</a></li>
</ul>
<p>Tutorials:</p>
<ul>
<li><a href="https://aws.amazon.com/premiumsupport/knowledge-center/eks-access-kubernetes-services/" rel="nofollow noreferrer">How do I provide external access to multiple Kubernetes services in my Amazon EKS cluster?</a></li>
<li><a href="https://v1-18.docs.kubernetes.io/docs/concepts/services-networking/ingress/" rel="nofollow noreferrer">ingress</a></li>
<li><a href="https://joachim8675309.medium.com/adding-ingress-with-amazon-eks-6c4379803b2" rel="nofollow noreferrer">Nginx Ingress with Amazon EKS</a></li>
</ul>
