<p>I got 2 spring app pods deployed in k8 cluster. 1 replica each. Both have their cluster-ip services exposing the services.</p>
<ol>
<li><em><strong>/upstream/agentLabs/makeotherTierCall</strong></em> should forward to prevlab deployment <em><strong>/agentLabs/makeOtherTierCall</strong></em></li>
<li><em><strong>/downstream/basics/hello</strong></em> should forward to newlab deployment <em><strong>/basics/hello</strong></em></li>
</ol>
<p>I am using ingress-nginx to redirect the traffic with following rules.</p>
<p><strong>ingress-service.yml</strong></p>
<pre><code>apiVersion: networking.k8s.io/v1
# UPDATE API
kind: Ingress
metadata:
  namespace: javaspace
  name: ingress-service
  annotations:
    kubernetes.io/ingress.class: 'nginx'
    nginx.ingress.kubernetes.io/use-regex: 'true'
    # ADD ANNOTATION
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    # UPDATE ANNOTATION
spec:
  rules:
    - http:
        paths:
          - path: /upstream?(.*)
            # UPDATE PATH
            pathType: Prefix
            # ADD PATHTYPE
            backend:
              service:
                # UPDATE SERVICE FIELDS
                name: prevlab-cluster-ip-service
                port:
                  number: 8080
          - path: /downstream?(.*)
            # UPDATE PATH
            pathType: Prefix
            # ADD PATHTYPE
            backend:
              service:
                # UPDATE SERVICE FIELDS
                name: newlab-cluster-ip-service
                port:
                  number: 8080
</code></pre>
<p>The issue is when I make any call (both 1 and 2) sometimes the ingress controller forwards the traffic correctly and sometimes it doesn't and I get 404. Basically, what I am observing is that alternatively the traffic is getting routed to both cluster-ip services one after the other.</p>
<p>for eg. if I make <em>/upstream/agentLabs/makeotherTierCall</em> request 2 times, at first it forwards wrongly to newlab cluster-ip service (giving back 404), and then next it forwards correctly to prevlab cluster-ip service (giving back 200).</p>
<p><strong>ingress-nginx controller setup</strong></p>
<pre><code>kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.0.1/deploy/static/provider/cloud/deploy.yaml
</code></pre>
<p><strong>ingress-controller logs</strong></p>
<pre><code>192.168.65.3 - - [24/Sep/2021:10:04:21 +0000] &quot;GET /upstream/agentLabs/makeOtherTierCall HTTP/2.0&quot; 200 12 &quot;-&quot; &quot;Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36&quot; 45 0.008 [javaspace-prevlab-cluster-ip-service-8080] [] 10.1.0.181:8080 12 0.007 200 24ae53531ce4d7109004f81e79534ca4
192.168.65.3 - - [24/Sep/2021:10:04:21 +0000] &quot;GET /upstream/agentLabs/makeOtherTierCall HTTP/2.0&quot; 404 286 &quot;-&quot; &quot;Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36&quot; 45 0.006 [javaspace-prevlab-cluster-ip-service-8080] [] 10.1.0.179:8080 286 0.005 404 1a32efb3a9dc21cce01b908afeb0248a
192.168.65.3 - - [24/Sep/2021:10:04:22 +0000] &quot;GET /upstream/agentLabs/makeOtherTierCall HTTP/2.0&quot; 200 12 &quot;-&quot; &quot;Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36&quot; 45 0.007 [javaspace-prevlab-cluster-ip-service-8080] [] 10.1.0.181:8080 12 0.007 200 542bd4abba25be5a416deca1152cb29b
192.168.65.3 - - [24/Sep/2021:10:04:22 +0000] &quot;GET /upstream/agentLabs/makeOtherTierCall HTTP/2.0&quot; 404 286 &quot;-&quot; &quot;Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36&quot; 45 0.008 [javaspace-prevlab-cluster-ip-service-8080] [] 10.1.0.179:8080 286 0.007 404 aaea8e3bf60dab81ef7454d51863a22d 

192.168.65.3 - - [24/Sep/2021:10:07:23 +0000] &quot;GET /downstream/basics/hello HTTP/2.0&quot; 404 306 &quot;-&quot; &quot;Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36&quot; 35 0.009 [javaspace-newlab-cluster-ip-service-8080] [] 10.1.0.181:8080 306 0.009 404 178bd79cc71b73f7a337a2652322d65f
192.168.65.3 - - [24/Sep/2021:10:07:23 +0000] &quot;GET /downstream/basics/hello HTTP/2.0&quot; 200 12 &quot;-&quot; &quot;Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36&quot; 35 0.002 [javaspace-newlab-cluster-ip-service-8080] [] 10.1.0.179:8080 12 0.003 200 280d2b7974ae38a467237cc6cf437b98
192.168.65.3 - - [24/Sep/2021:10:07:24 +0000] &quot;GET /downstream/basics/hello HTTP/2.0&quot; 404 306 &quot;-&quot; &quot;Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36&quot; 35 0.008 [javaspace-newlab-cluster-ip-service-8080] [] 10.1.0.181:8080 306 0.008 404 4e237232b4c53c9bb90e7bb59c15e916
192.168.65.3 - - [24/Sep/2021:10:07:24 +0000] &quot;GET /downstream/basics/hello HTTP/2.0&quot; 200 12 &quot;-&quot; &quot;Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36&quot; 35 0.003 [javaspace-newlab-cluster-ip-service-8080] [] 10.1.0.179:8080 12 0.002 200 be0cf532a972ec0d783e15869470f579
</code></pre>
<p>Using Docker Desktop's Kubernetes. Kubernetes 1.21.2</p>
<p><strong>What am I doing wrong here? Is this default behavior? Do I need to do some additional config?</strong></p>

<p>Nevermind. I had done a very small but very stupid mistake. In case someone comes wandering around here: I had used the same label for my 2 backend services. And for each of the cluster-IP services used the same selector. Hence, I was seeing inconsistent behavior.</p>
