<p>We have confluent kafka 4.0.0 used in our env as a microservice, we would like to get it upgraded to 5.5.1 version similarly the zookeeper and kafka Rest.
We are running kafka broker and zookeeper as statefulsets and the kafka rest as deployment.</p>
<p>What is the best way of doing this upgrade with rollback procedures on?</p>
<blockquote>
<ol>
<li>Upgrading the image tag to the version 5.5.1 on the running sts and deployments and change all the configurations under the same
namespace.</li>
</ol>
</blockquote>
<p>or</p>
<blockquote>
<ol start="2">
<li>Creating all the kafka components in a new namespace with upgraded version and new configuration on it and simply route the traffic here
by changing the ingress load balancer.</li>
</ol>
</blockquote>
<p>What would be the best way of upgrading out of these methods? also let me know if there is any other way of doing it.</p>

<p>We're doing the 2nd way:</p>
<ol>
<li>Spin up the new kafka</li>
<li>Spin up the new set of workers listening to the new kafka</li>
<li>Inject message to test workers-kafka integration</li>
<li>Spin up the new app stack feeding the new kafka</li>
<li>Inject traffic to test app-kafka-workers integration</li>
<li>Switch traffic to the new stack</li>
</ol>
<p>This is much safer than upgrading live as it lets you verify/debug on all the stages.</p>
