<p>Yesterday I was testing having a production and staging deployment running at the same time. The results I got were erratic in that sometimes you'd go to the staging URL and it would load the production. If you refreshed the page, it would switch between the two randomly.</p>

<p>I won't have time to test it until this weekend because QA is going on, and this issue disrupted it from happening. I killed the production deployment and removed the routes from my <code>ingress.yaml</code> so QA could continue without issue.</p>

<p>At any rate, my <code>ingress.yaml</code> configuration was likely the cause, so wanted to share it to see what caused this behavior:</p>

<pre><code>apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/add-base-url: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.org/client-max-body-size: "500m"
    nginx.ingress.kubernetes.io/use-regex: "true"
  name: ingress-service
  namespace: default
spec:
  tls:
    - hosts:
        - domain.com
        - www.domain.com
        - staging.domain.com
      secretName: tls-domain-com
  rules:
    - host: domain.com
      http:
        paths:
          - path: /?(.*)
            backend:
              serviceName: client-cluster-ip-service-prod
              servicePort: 3000
          - path: /admin/?(.*)
            backend:
              serviceName: admin-cluster-ip-service-prod
              servicePort: 4000
          - path: /api/?(.*)
            backend:
              serviceName: api-cluster-ip-service-prod
              servicePort: 5000
    - host: www.domain.com
      http:
        paths:
          - path: /?(.*)
            backend:
              serviceName: client-cluster-ip-service-prod
              servicePort: 3000
          - path: /admin/?(.*)
            backend:
              serviceName: admin-cluster-ip-service-prod
              servicePort: 4000
          - path: /api/?(.*)
            backend:
              serviceName: api-cluster-ip-service-prod
              servicePort: 5000
    - host: staging.domain.com
      http:
        paths:
          - path: /?(.*)
            backend:
              serviceName: client-cluster-ip-service-staging
              servicePort: 3000
          - path: /admin/?(.*)
            backend:
              serviceName: admin-cluster-ip-service-staging
              servicePort: 4000
          - path: /api/?(.*)
            backend:
              serviceName: api-cluster-ip-service-staging
              servicePort: 5000

</code></pre>

<p>I have feeling it is one of the following:</p>

<ul>
<li>Having the <code>domain.com</code> before the others and that it should be at the end</li>
<li>Now that I'm looking at it again, they are at the same IP and using the same ports, so the ports need to be changed</li>
<li>If I want to leave the ports the same, I'd need to deploy another ingress controller having one for staging and one for production and follow <a href="https://kubernetes.github.io/ingress-nginx/user-guide/multiple-ingress/" rel="nofollow noreferrer">these instructions</a></li>
</ul>

<p>At any rate, can anyone confirm?</p>

<p><strong>EDIT</strong></p>

<p>Adding <code>.yaml</code>:</p>

<pre><code># client-staging.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: client-deployment-staging
spec:
  replicas: 3
  selector:
    matchLabels:
      component: client
  template:
    metadata:
      labels:
        component: client
    spec:
      containers:
        - name: client
          image: testappacr.azurecr.io/test-app-client
          ports:
            - containerPort: 3000
          env:
          - name: DOMAIN
            valueFrom:
              secretKeyRef:
                name: test-app-staging-secrets
                key: DOMAIN 
---
apiVersion: v1
kind: Service
metadata:
  name: client-cluster-ip-service-staging
spec:
  type: ClusterIP
  selector:
    component: client
  ports:
    - port: 3000
      targetPort: 3000
</code></pre>

<pre><code># client-prod.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: client-deployment-prod
spec:
  replicas: 3
  selector:
    matchLabels:
      component: client
  template:
    metadata:
      labels:
        component: client
    spec:
      containers:
        - name: client
          image: testappacr.azurecr.io/test-app-client
          ports:
            - containerPort: 3000
          env:
          - name: DOMAIN
            valueFrom:
              secretKeyRef:
                name: test-app-prod-secrets
                key: DOMAIN 

---
apiVersion: v1
kind: Service
metadata:
  name: client-cluster-ip-service-prod
spec:
  type: ClusterIP
  selector:
    component: client
  ports:
    - port: 3000
      targetPort: 3000
</code></pre>

cjones
<p>Without seeing the deployments yaml descriptors and by the description of the problem the most probable cause is that both deployments are getting in the service loadbalancers endpoints, so you must change the selectors and add something like: <code>env: prod</code> and <code>env: staging</code> and add those to each deployment.</p>

<p>For checking if this is the issue run a <code>kubectl describe service</code> for each service and check the endpoints.</p>

<p>If not, let me know the output and i can help you debug further.</p>

<p>EDIT: Changes in files after posting them:</p>

<h1>Production</h1>

<p>Service:</p>

<pre><code>spec:
  type: ClusterIP
  selector:
    component: client
    environment: production
</code></pre>

<p>Deployment:</p>

<pre><code>  replicas: 3
  selector:
    matchLabels:
      component: client
      environment: production
  template:
    metadata:
      labels:
        component: client
        environment: production
</code></pre>

<h1>Staging</h1>

<p>Service:</p>

<pre><code>spec:
  type: ClusterIP
  selector:
    component: client
    environment: staging
</code></pre>

<p>Deployment:</p>

<pre><code>  replicas: 3
  selector:
    matchLabels:
      component: client
      environment: staging
  template:
    metadata:
      labels:
        component: client
        environment: staging
</code></pre>
