<p>I am running a 3 nodes bare metal cluster on version <code>1.9.5</code>.</p>

<p>IPs of the 3 nodes are :</p>

<pre><code>[root@node1 new]# kubectl get nodes
NAME      STATUS    ROLES         AGE       VERSION    IP
node1     Ready     master,node   1d        v1.9.5     172.16.16.1
node2     Ready     node          1d        v1.9.5     172.16.16.2
node3     Ready     node          1d        v1.9.5     172.16.16.3
</code></pre>

<p>Everything I explain below is being done in 1 namespace i.e. <code>ingress-nginx</code></p>

<p>I have 2 apps deployed.</p>

<pre><code>[root@node1 new]# kubectl get po -n ingress-nginx
NAME                                     READY     STATUS    RESTARTS   AGE
app1-5d4d466cc7-595lc                    1/1       Running   0          25m
app2-55cf97d86d-9v8gl                    1/1       Running   0          25m
</code></pre>

<p>Their services are :</p>

<pre><code>[root@node1 new]# kubectl get svc -n ingress-nginx
NAME                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                        AGE
appsvc1                NodePort    10.233.60.145   &lt;none&gt;        80:32601/TCP                   25m
appsvc2                NodePort    10.233.46.230   &lt;none&gt;        80:30616/TCP                   25m
</code></pre>

<p>So when I access them via <code>NodePort</code>, I get my desired result.</p>

<pre><code>curl http://172.16.16.2:32601
&lt;h1&gt;Hello app1!&lt;/h1&gt;

curl http://172.16.16.2:30616
&lt;h1&gt;Hello app2!&lt;/h1&gt;
</code></pre>

<p>Now my aim is to configure path based routing using nginx ingress controller so that at the end of it, I can get routing to app1 using</p>

<pre><code>curl http://172.16.16.2/app1
&lt;h1&gt;Hello app1!&lt;/h1&gt;

&amp;

curl http://172.16.16.2/app2
&lt;h1&gt;Hello app2!&lt;/h1&gt;
</code></pre>

<p>So now I have setup an ingress controller using <a href="https://github.com/kubernetes/ingress-nginx/tree/master/deploy" rel="nofollow noreferrer">ingress-nginx</a> .</p>

<p>The nginx controller is also deployed in the same namespace at the apps i.e. <code>ingress-nginx</code></p>

<p>My ingress controller is successfully deployed as the logs indicate :</p>

<pre><code>[root@node1 new]# kubectl logs nginx-ingress-controller-9c7b694-bjn6h -n ingress-nginx
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:    0.12.0
  Build:      git-1df421a
  Repository: https://github.com/kubernetes/ingress-nginx
-------------------------------------------------------------------------------

I0415 16:08:18.736790       5 main.go:225] Running in Kubernetes Cluster version v1.9 (v1.9.5) - git (clean) commit f01a2bf98249a4db383560443a59bed0c13575df - platform linux/amd64
I0415 16:08:18.743855       5 main.go:84] validated ingress-nginx/default-http-backend as the default backend
I0415 16:08:19.182913       5 stat_collector.go:77] starting new nginx stats collector for Ingress controller running in namespace  (class nginx)
I0415 16:08:19.182944       5 stat_collector.go:78] collector extracting information from port 18080
I0415 16:08:19.211749       5 nginx.go:281] starting Ingress controller
I0415 16:08:20.325839       5 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"app-ingress", UID:"918405ac-410e-11e8-a473-080027917402", APIVersion:"extensions", ResourceVersion:"75539", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress default/app-ingress
I0415 16:08:20.326503       5 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"ingress-nginx", Name:"nginx-ingress", UID:"8dc22500-410e-11e8-a473-080027917402", APIVersion:"extensions", ResourceVersion:"75538", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress ingress-nginx/nginx-ingress
I0415 16:08:20.413514       5 store.go:614] running initial sync of secrets
I0415 16:08:20.413591       5 nginx.go:302] starting NGINX process...
I0415 16:08:20.418356       5 leaderelection.go:174] attempting to acquire leader lease...
W0415 16:08:20.422596       5 controller.go:777] service ingress-nginx/nginx-ingress does not have any active endpoints
I0415 16:08:20.422686       5 controller.go:183] backend reload required
I0415 16:08:20.422694       5 stat_collector.go:34] changing prometheus collector from  to default
I0415 16:08:20.439620       5 status.go:196] new leader elected: nginx-ingress-controller-9c7b694-h2n4b
I0415 16:08:20.534277       5 controller.go:192] ingress backend successfully reloaded...
W0415 16:08:28.768140       5 controller.go:777] service ingress-nginx/nginx-ingress does not have any active endpoints
I0415 16:09:00.478068       5 leaderelection.go:184] successfully acquired lease ingress-nginx/ingress-controller-leader-nginx
I0415 16:09:00.478207       5 status.go:196] new leader elected: nginx-ingress-controller-9c7b694-bjn6h
</code></pre>

<p>Then I configured my ingress using :</p>

<pre><code>apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
  name: app-ingress
spec:
  rules:
  - host: my-test.com
    http:
      paths:
      - backend:
          serviceName: appsvc1
          servicePort: 80
        path: /app1
      - backend:
          serviceName: appsvc2
          servicePort: 80
        path: /app2
</code></pre>

<p>I created this using :</p>

<pre><code>kubectl create -f app-ingress.yaml -n ingress-nginx
</code></pre>

<p>I then expose this using :</p>

<pre><code>apiVersion: v1
kind: Service
metadata:
  name: nginx-ingress
spec:
  type: NodePort
  externalIps:
  - 172.16.16.1
  - 172.16.16.2
  - 172.16.16.3
  ports:
    - port: 80
      nodePort: 30000
      name: http
  selector:
    app: nginx-ingress
</code></pre>

<p><code>app: nginx-ingress</code> points to label from my label in ingress-controller pod.</p>

<p>I deploy it using :</p>

<pre><code>kubectl create -f nginx-ingress-controller-service.yaml -n=ingress
</code></pre>

<p>But when I try to access the apps using the URLs, I get :</p>

<pre><code>curl http://172.16.16.2/app1
default backend - 404

&amp;

curl http://172.16.16.2/app2
default backend - 404
</code></pre>

<p>Even doing </p>

<pre><code>curl http://my-test.com/app1
default backend - 404

&amp;

curl http://my-test.com/app2
default backend - 404
</code></pre>

<p>does not work.</p>

<p>My /etc/hosts files is :</p>

<pre><code>172.16.16.1 my-test.com
</code></pre>

<p>Is there something I have missed or am doing wrong ? </p>

<pre><code>[root@node1 new]# kubectl get all -n ingress-nginx
NAME                              DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deploy/app1                       2         2         2            2           48m
deploy/app2                       2         2         2            2           48m
deploy/default-http-backend       1         1         1            1           3h
deploy/nginx-ingress-controller   1         1         1            1           3h

NAME                                  DESIRED   CURRENT   READY     AGE
rs/app1-5d4d466cc7                    2         2         2         48m
rs/app2-55cf97d86d                    2         2         2         48m
rs/default-http-backend-55c6c69b88    1         1         1         3h
rs/nginx-ingress-controller-9c7b694   1         1         1         3h

NAME                              DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deploy/app1                       2         2         2            2           48m
deploy/app2                       2         2         2            2           48m
deploy/default-http-backend       1         1         1            1           3h
deploy/nginx-ingress-controller   1         1         1            1           3h

NAME                                  DESIRED   CURRENT   READY     AGE
rs/app1-5d4d466cc7                    2         2         2         48m
rs/app2-55cf97d86d                    2         2         2         48m
rs/default-http-backend-55c6c69b88    1         1         1         3h
rs/nginx-ingress-controller-9c7b694   1         1         1         3h

NAME                                        READY     STATUS    RESTARTS   AGE
po/app1-5d4d466cc7-595lc                    1/1       Running   0          48m
po/app1-5d4d466cc7-5dn72                    1/1       Running   0          48m
po/app2-55cf97d86d-9v8gl                    1/1       Running   0          48m
po/app2-55cf97d86d-lckpn                    1/1       Running   0          48m
po/default-http-backend-55c6c69b88-8shkt    1/1       Running   0          3h
po/nginx-ingress-controller-9c7b694-bjn6h   1/1       Running   0          52m

NAME                       TYPE        CLUSTER-IP      EXTERNAL-IP                           PORT(S)                      AGE
svc/appsvc1                NodePort    10.233.60.145   &lt;none&gt;                                80:32601/TCP                 48m
svc/appsvc2                NodePort    10.233.46.230   &lt;none&gt;                                80:30616/TCP                 48m
svc/default-http-backend   ClusterIP   10.233.5.30     &lt;none&gt;                                80/TCP                       3h
svc/ingress-nginx          NodePort    10.233.6.186    &lt;none&gt;                                80:31301/TCP,443:32103/TCP   3h
svc/nginx-ingress          NodePort    10.233.9.163    172.16.16.1,172.16.16.2,172.16.16.3   80:30000/TCP                 2h
</code></pre>

<p>I even changed my ingress config to remove the host and be open for all hosts:</p>

<pre><code>apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
  name: app-ingress
spec:
  rules:
  - http:
      paths:
      - path: /app1
        backend:
          serviceName: appsvc1
          servicePort: 80
      - path: /app2
        backend:
          serviceName: appsvc2
          servicePort: 80
</code></pre>

<p>And now I get redirected.</p>

<pre><code>[root@node1 ~]# curl http://172.16.16.1/app1
&lt;html&gt;
&lt;head&gt;&lt;title&gt;308 Permanent Redirect&lt;/title&gt;&lt;/head&gt;
&lt;body bgcolor="white"&gt;
&lt;center&gt;&lt;h1&gt;308 Permanent Redirect&lt;/h1&gt;&lt;/center&gt;
&lt;hr&gt;&lt;center&gt;nginx/1.13.9&lt;/center&gt;
&lt;/body&gt;
&lt;/html&gt;
[root@node1 ~]# curl http://172.16.16.1/app2
&lt;html&gt;
&lt;head&gt;&lt;title&gt;308 Permanent Redirect&lt;/title&gt;&lt;/head&gt;
&lt;body bgcolor="white"&gt;
&lt;center&gt;&lt;h1&gt;308 Permanent Redirect&lt;/h1&gt;&lt;/center&gt;
&lt;hr&gt;&lt;center&gt;nginx/1.13.9&lt;/center&gt;
&lt;/body&gt;
&lt;/html&gt;
</code></pre>

<p>Ingress controller logs say :</p>

<pre><code>10.233.102.128 - [10.233.102.128] - - [15/Apr/2018:17:35:52 +0000] "GET /app1 HTTP/1.1" 308 187 "-" "curl/7.29.0" 80 0.000 [ingress-nginx-appsvc1-80] - - - -
10.233.102.128 - [10.233.102.128] - - [15/Apr/2018:17:36:09 +0000] "GET /app1 HTTP/1.1" 308 187 "-" "curl/7.29.0" 79 0.000 [ingress-nginx-appsvc1-80] - - - -
10.233.102.128 - [10.233.102.128] - - [15/Apr/2018:17:36:11 +0000] "GET /app2 HTTP/1.1" 308 187 "-" "curl/7.29.0" 79 0.000 [ingress-nginx-appsvc2-80] - - - -
10.233.102.128 - [10.233.102.128] - - [15/Apr/2018:17:36:25 +0000] "GET /app2 HTTP/1.1" 308 187 "-" "curl/7.29.0" 85 0.000 [ingress-nginx-appsvc2-80] - - - -
10.233.102.128 - [10.233.102.128] - - [15/Apr/2018:17:36:51 +0000] "GET /app2 HTTP/1.1" 308 187 "-" "curl/7.29.0" 80 0.000 [ingress-nginx-appsvc2-80] - - - -
</code></pre>

<p>All your configs are fine, except one thing related to annotation:</p>

<pre><code>  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
</code></pre>

<p>You don't need this annotation, because it's processed earlier than routing and you always have the request to <code>/</code> . You have no route for this path, so just remove this annotation and it should work. </p>
