<p>Here is an example cluster setup that matches my production cluster:</p>

<pre><code>Ingress rule abc.com &gt; Service-AB &gt; Pod-A, Pod-B &gt; cluster-A, cluster-B &gt; image-0.1.1

Ingress rule xyz.com &gt; Service-CD &gt; Pod-C, Pod-D &gt; cluster-C, cluster-D &gt; image-1.3.2
</code></pre>

<p>Imagine I need to update cluster1's image to <code>v0.1.2</code>, to do this I can just change the deployment manifest and <code>apply</code> it to perform a rolling update but before that I want to do an e2e test in production cluster to make sure the new image, behaves as expected there.</p>

<p>To accomplish this, I create a <a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.14/#pod-v1-core" rel="nofollow noreferrer">Pod Spec</a> having the new image version together with a new service that selects only that new pod and an ingress rule to reach this new service. It looks like following,</p>

<pre><code>Ingress rule abc.com/update &gt; Service-Z &gt; Pod-Z &gt; cluster-Z &gt; image-0.1.2
</code></pre>

<p>Now I point my e2e test to the endpoint <code>abc.com/update</code>, once successful, I delete the new ingress rule, service and pod and follow the normal process of rolling update by updating the deployment with new container image version and performing an <code>apply</code>.</p>

<p>Is it a correct way to do testing in production cluster, can there be a better (easier) way?</p>

Praveen Tiwari
<p>I think it varies depending on the technology that you are using. Given your case, which is using plain kubernetes. I would use a proxy in between to do the A/B testing. Here is the description:</p>

<p><strong>Before upgrade:</strong><br>
Ingress rule abc.com -> svc-proxy -> pod-proxy --> svc-my-pod (v1.0.0)</p>

<p><strong>After upgrade (A/B) testing:</strong> <br>
Ingress rule abc.com -> svc-proxy -> pod-proxy --> svc-my-pod-1 (v1.0.0)<br>
&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp; | <br>
&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&ensp;&emsp;--> svc-my-pod-2 (v1.1.0)</p>

<p><strong>Fully upgraded</strong><br>
Ingress rule abc.com -> svc-proxy -> pod-proxy -> my-pod (v1.1.0)
<br>
<br>
<br>
Note that in the A/B testing phase you would have to add an upstream with weights (if using nginx) to emulate the desired weighted policy.</p>

<pre><code>upstream dynamic {
    server pod-proxy-1      weight=2;
    server pod-proxy-2      weight=4;
}

server {
    location / {
        proxy_pass http://dynamic;
    }
}
</code></pre>

<h2>Using services and endpoints to switch between pods</h2>

<p>Have a yaml with v1 called pod-v1. You would define its match label like the following yaml:</p>

<pre><code>spec:
  selector:
    matchLabels:
      app: pod-v1
</code></pre>

<p>Have a yaml with v2 called pod-v2. You would define its match label like the following yaml:</p>

<pre><code>spec:
  selector:
    matchLabels:
      app: pod-v2
</code></pre>

<p>Then you can easily change routing between the pods by changing the selector of a service. Like the following yaml:</p>

<pre><code>apiVersion: v1
kind: Service
metadata:
  name: pod
  labels:
    app: pod
spec:
  ports:
  - name: "8000"
    port: 8000
    targetPort: 8000
  selector:
    app: pod-v1
status:
  loadBalancer: {}
</code></pre>

<p>Note that to route the traffic to v2 you would just change the selector to match the label pod-v2.</p>

<pre><code>selector:
  app: pod-v2
</code></pre>
