<p>How to do a canary upgrade to existing istio customised setup.</p>
<p><strong>Requirements:</strong></p>
<ul>
<li>We have existing customised setup of istio 1.7.3 (installed using istoctl method and no revision set for this) for AKS 1.18.14.</li>
<li>Now we need to upgrade to istio 1.8 with no downtime or minimal.</li>
<li>The upgrade should be safer and it wont break our prod environemnt in any ways.</li>
</ul>
<p>How we installed the current istio customised environment:</p>
<ol>
<li>created manifest.</li>
</ol>
<pre class="lang-text prettyprint-override"><code>istioctl manifest generate --set profile=default -f /manifests/overlay/overlay.yaml &gt; $HOME/generated-manifest.yaml
</code></pre>
<ol start="2">
<li>installed istio.</li>
</ol>
<pre class="lang-text prettyprint-override"><code>istioctl install --set profile=default -f /manifests/overlay/overlay.yaml
</code></pre>
<ol start="3">
<li>Verified istio against the deployed manifest.</li>
</ol>
<pre class="lang-text prettyprint-override"><code>istioctl verify-install -f $HOME/generated-manifest.yaml
</code></pre>
<h3>Planned upgrade process <sup><a href="https://istio.io/latest/docs/setup/upgrade/canary/" rel="nofollow noreferrer">Reference</a></sup></h3>
<ol>
<li>Precheck for upgrade.</li>
</ol>
<pre class="lang-text prettyprint-override"><code>istioctl x precheck
</code></pre>
<ol start="2">
<li>export the current used configuration of istio using below command to a yaml file.</li>
</ol>
<pre class="lang-text prettyprint-override"><code>kubectl -n istio-system get iop installed-state-install -o yaml &gt; /tmp/iop.yaml
</code></pre>
<ol start="3">
<li>Download istio 1.8 binary and extract the directory and navigate the directory to where we have the 1.8 version istioctl binary.</li>
</ol>
<pre class="lang-text prettyprint-override"><code>cd istio1.8\istioctl1.8
</code></pre>
<ol start="4">
<li>from the new version istio directory, create a new controlplane for istio(1.8) with proper revision set and use the previously exported installed-state &quot;iop.yaml&quot;.</li>
</ol>
<pre class="lang-text prettyprint-override"><code>./istioctl1.8 install --set revision=1-8 --set profile=default -f /tmp/iop.yaml
</code></pre>
<p>Expect that it will create new control plane with existing
costamised configuration and now we will have two control plane
deployments and services running side-by-side:</p>
<pre class="lang-text prettyprint-override"><code>kubectl get pods -n istio-system -l app=istiod
NAME                                    READY   STATUS    RESTARTS   AGE
istiod-786779888b-p9s5n                 1/1     Running   0          114m
istiod-1-7-6956db645c-vwhsk             1/1     Running   0          1m 
</code></pre>
<ol start="5">
<li>After this, we need to change the existing label of all our cluster namespaces where we need to inject the istio proxy containers. Need to remove the old &quot;istio-injection&quot; label, and add the istio.io/rev label to point to the canary revision 1-8.</li>
</ol>
<pre class="lang-text prettyprint-override"><code>kubectl label namespace test-ns istio-injection- istio.io/rev=1-8
</code></pre>
<p>Hope, at this point also the environment is stable with old istio configurations and we can make decision on which app pods can be restarted to make the new control plane changes as per our downtime, and its allowed to run some apps with older control plane and another with new controller plane configs t this point. eg:</p>
<pre class="lang-text prettyprint-override"><code>kubectl rollout restart deployment -n test-ns  (first) 
kubectl rollout restart deployment -n test-ns2 (later)
kubectl rollout restart deployment -n test-ns3 (again after sometieme later)
</code></pre>
<ol start="6">
<li>Once we planed for downtime and restarted the deployments as we decided, confirm all the pods are now using dataplane proxy injector of version 1.8 only</li>
</ol>
<pre class="lang-text prettyprint-override"><code>kubectl get pods -n test-ns -l istio.io/rev=1-8
</code></pre>
<ol start="7">
<li>To verify that the new pods in the test-ns namespace are using the istiod-canary service corresponding to the canary revision</li>
</ol>
<pre class="lang-text prettyprint-override"><code>istioctl proxy-status | grep ${pod_name} | awk '{print $7}'
</code></pre>
<ol start="8">
<li>After upgrading both the control plane and data plane, can uninstall the old control plane</li>
</ol>
<pre class="lang-text prettyprint-override"><code>istioctl x uninstall -f /tmp/iop.yaml.
</code></pre>
<h3>Need to clear below points before upgrade.</h3>
<ol>
<li><p>Are all the steps prepared for the upgrade above are good to proceed for highly used Prod environment?</p>
</li>
<li><p>By exporting the installed state iop is enough to get all customised step to proceed the canary upgrade? or is there any chance of braking the upgrade or missing any settings?</p>
</li>
<li><p>Whether the step 4 above will create the 1.8 istio control plane with all the customization as we already have without any break or missing something?</p>
</li>
<li><p>after the step 4, do we need to any any extra configuration related to istiod service configuration&gt; the followed document is not clear about that,</p>
</li>
<li><p>for the step 5 above, how we can identy all the namespaces, where we have the istio-injection enabled and only modify those namespace alone and leave others as it was before?</p>
</li>
<li><p>so for the step 8 above, how to ensure we are uninstalling old control plane only ? we have to get the binary for old controlplane say (1.7 in my case)and use that binary with same exported <code>/tmp/iop.yaml</code>?</p>
</li>
<li><p>No Idea about how to rollback any issues happened in between.. before or after the old controlplane deleted</p>
</li>
</ol>

<ol>
<li><p>No. You should go through <a href="https://istio.io/latest/news/releases/1.8.x/announcing-1.8/change-notes/" rel="nofollow noreferrer">changelog</a> and <a href="https://istio.io/latest/news/releases/1.8.x/announcing-1.8/upgrade-notes/" rel="nofollow noreferrer">upgrade notes</a>. See what's new, what's changed, depracted etc. Adjust your configs accordingly.</p>
</li>
<li><p>In theory - yes, in practice - no. See above. That's why you should always check upgarde notes/changelog and plan accordingly. There is always a slim chance something will go wrong.</p>
</li>
<li><p>It should, but again, be prepared that something may break (One more time - go through changelog/upgrade notes, this is important).</p>
</li>
<li><p>No.</p>
</li>
<li><p>You can find all namespaces with Istio injection enabled with:</p>
</li>
</ol>
<pre class="lang-text prettyprint-override"><code>kubectl get namespaces -l=istio-injection=enabled
</code></pre>
<p>Istio upgrade process should only modify namespaces with injection enabled (and <code>istio-system</code> namespace).</p>
<ol start="6">
<li>If your old control plane does not have a <code>revision</code> label, you have to uninstall it using its original installation options (old yaml file)</li>
</ol>
<pre class="lang-text prettyprint-override"><code>istioctl x uninstall -f /path/to/old/config.yaml
</code></pre>
<p>If it does have <code>revision</code> label:</p>
<pre class="lang-text prettyprint-override"><code>istioctl x uninstall --revision &lt;revision&gt;
</code></pre>
<ol start="7">
<li>You can just uninstall new control plane with</li>
</ol>
<pre class="lang-text prettyprint-override"><code>istioctl x uninstall revision=1-8
</code></pre>
<p>This will revert to the old control plane, assuming you have not yet uninstalled it. However, you will have to reinstall gateways for the old version manually, as the uninstall command does not revert them automatically.</p>
<hr />
<p>I would strongly recommend creating a temporary test environment. Recreating existing cluster on test env. Performing upgrade there, and adjusting the process to meet your needs.<br />
This way you will avoid catastrofic failures on your production environment.</p>
