<p>My kubernetes application is made of several flavors of nodes, a couple of “schedulers” which send tasks to quite a few more “worker” nodes. In order for this app to work correctly all the nodes must be of exactly the same code version.</p>
<p>The deployment is performed using a standard ReplicaSet and when my CICD kicks in it just does a simple rolling update. This causes a problem though since during the rolling update, nodes of different code versions co-exist for a few seconds, so a few tasks during this time get wrong results.</p>
<p>Ideally what I would want is that deploying a new version would create a completely new application that only communicates with itself and has time to warm its cache, then on a flick of a switch this new app would become active and start to get new client requests. The old app would remain active for a few more seconds and then shut down.</p>
<p>I’m using Istio sidecar for mesh communication.</p>
<p>Is there a standard way to do this? How is such a requirement usually handled?</p>

<p>I also had such a situation. Kubernetes alone cannot satisfy your requirement, I was also not able to find any tool that allows to coordinate multiple deployments together (although <a href="https://flagger.app/" rel="nofollow noreferrer">Flagger</a> looks promising).</p>
<p>So the only way I found was by using CI/CD: Jenkins in my case. I don't have the code, but the idea is the following:</p>
<ol>
<li><p>Deploy all application deployments using single Helm chart. Every Helm release name and corresponding Kubernetes labels must be based off of some sequential number, e.g. Jenkins <code>$BUILD_NUMBER</code>. Helm release can be named like <code>example-app-${BUILD_NUMBER}</code> and all deployments must have label <code>version: $BUILD_NUMBER</code> . Important part here is that your <code>Services</code> should not be a part of your Helm chart because they will be handled by Jenkins.</p>
</li>
<li><p>Start your build with detecting the current version of the app (using bash script or you can store it in <code>ConfigMap</code>).</p>
</li>
<li><p>Start <code>helm install example-app-{$BUILD_NUMBER}</code> with <code>--atomic</code> flag set. Atomic flag will make sure that the release is properly removed on failure. And don't delete previous version of the app yet.</p>
</li>
<li><p>Wait for Helm to complete and in case of success run <code>kubectl set selector service/example-app version=$BUILD_NUMBER</code>. That will instantly switch Kubernetes <code>Service</code> from one version to another. If you have multiple services you can issue multiple <code>set selector</code> commands (each command executes immediately).</p>
</li>
<li><p>Delete previous Helm release and optionally update <code>ConfigMap</code> with new app version.</p>
</li>
</ol>
<p>Depending on your app you may want to run tests on non user facing <code>Services</code> as a part of step 4 (after Helm release succeeds).</p>
<p>Another good idea is to have <a href="https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/" rel="nofollow noreferrer"><code>preStop</code></a> hooks on your worker pods so that they can finish their jobs before being deleted.</p>
