<p>Application is deployed on K8s using StatefulSet because of stateful in nature. There is around 250+ pods are running and HPA has been implemented on it too that can scale upto 400 pods.</p>
<p>When new deployment occurs, it takes longer time (~ 10-15m) to update all pods in Rolling Update fashion.</p>
<p><strong>Problem:</strong> End user get response from 2 version of pods until all pods are replaced with new revision.</p>
<p>I am googling for an architecture where overall deployment time can be reduced and getting the best possible solutions to use <code>BLUE/GREEN</code> strategy but it has bunch of impact with integrated services like monitoring, logging, telemetry etc because of 2 naming conventions.</p>
<p>Ideally I am looking for a solutions like <code>maxSurge</code> for Deployment in which firstly new pods are created and then traffic are shifted to it at a time but in case of StatefulSet, it won't support maxSurge with RollingUpdate strategy &amp; controller will delete and recreate each Pod in the StatefulSet based on ordinal index from bigger to smaller.</p>

Ashish Kumar
<p>The solution is to do a <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/statefulset#partitioning_rolling_updates" rel="nofollow noreferrer">partitioning rolling update</a> along with a <a href="https://cloud.google.com/architecture/implementing-deployment-and-testing-strategies-on-gke#perform_a_canary_test" rel="nofollow noreferrer">canary deployment</a>.</p>
<p>Let’s suppose we have the statefulset workload defined by the following yaml file:</p>
<pre><code>apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
    version: &quot;1.20&quot;
spec:
  ports:
  - port: 80
    name: web
  clusterIP: None
  selector:
    app: nginx
    version: &quot;1.20&quot;
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
spec:
  selector:
    matchLabels:
      app: nginx # Label selector that determines which Pods belong to the StatefulSet
                 # Must match spec: template: metadata: labels
  serviceName: &quot;nginx&quot;
  replicas: 3
  template:
    metadata:
      labels:
        app: nginx # Pod template's label selector
        version: &quot;1.20&quot;
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: nginx
        image: nginx:1.20
        ports:
        - containerPort: 80
          name: web
        volumeMounts:
        - name: www
          mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
  - metadata:
      name: www
    spec:
      accessModes: [ &quot;ReadWriteOnce&quot; ]
      resources:
        requests:
          storage: 1Gi
</code></pre>
<p>You could patch the statefulset to create a partition, and change the image and version label for the remaining pods: (In this case, since there are only 3 pods, the last one will be the one that will change its image.)</p>
<pre><code>$ kubectl patch statefulset web -p '{&quot;spec&quot;:{&quot;updateStrategy&quot;:{&quot;type&quot;:&quot;RollingUpdate&quot;,&quot;rollingUpdate&quot;:{&quot;partition&quot;:2}}}}'
$ kubectl patch statefulset web --type='json' -p='[{&quot;op&quot;: &quot;replace&quot;, &quot;path&quot;: &quot;/spec/template/spec/containers/0/image&quot;, &quot;value&quot;:&quot;nginx:1.21&quot;}]'
$ kubectl patch statefulset web --type='json' -p='[{&quot;op&quot;: &quot;replace&quot;, &quot;path&quot;: &quot;/spec/template/metadata/labels/version&quot;, &quot;value&quot;:&quot;1.21&quot;}]'

</code></pre>
<p>At this point, you have a pod with the new image and version label ready to use, but since the version label is different, the traffic is still going to the other two pods. If you change the version in the yaml file and apply the new configuration, the rollout will be transparent, since there is already a pod ready to migrate the traffic:</p>
<pre><code>apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
    version: &quot;1.21&quot;
spec:
  ports:
  - port: 80
    name: web
  clusterIP: None
  selector:
    app: nginx
    version: &quot;1.21&quot;
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
spec:
  selector:
    matchLabels:
      app: nginx # Label selector that determines which Pods belong to the StatefulSet
                 # Must match spec: template: metadata: labels
  serviceName: &quot;nginx&quot;
  replicas: 3
  template:
    metadata:
      labels:
        app: nginx # Pod template's label selector
        version: &quot;1.21&quot;
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: nginx
        image: nginx:1.21
        ports:
        - containerPort: 80
          name: web
        volumeMounts:
        - name: www
          mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
  - metadata:
      name: www
    spec:
      accessModes: [ &quot;ReadWriteOnce&quot; ]
      resources:
        requests:
          storage: 1Gi
</code></pre>
<pre><code>$ kubectl apply -f file-name.yaml
</code></pre>
<p>Once traffic is migrated to the pod containing the new image and version label, you should patch again the statefulset and remove the partition with the command <code>kubectl patch statefulset web -p '{&quot;spec&quot;:{&quot;updateStrategy&quot;:{&quot;type&quot;:&quot;RollingUpdate&quot;,&quot;rollingUpdate&quot;:{&quot;partition&quot;:0}}}}'</code></p>
<p>Note: You will need to be very careful with the size of the partition, since the remaining pods will handle the whole traffic for some time.</p>
