<p>I have a Kafka cluster hosted in GKE. Google updates GKE nodes in weekly basis and whenever this happens Kafka becomes temporarily unavailable and this causes massive error/rebalance to get it backup to healthy state. Currently we rely on K8 retry to eventually succeed once upgrade completes and cluster becomes available. Is there a way to gracefully handle this type of situation in Kafka or avoid it if possible?</p>

user_1357
<p>In order to be able to inform you better, you would have to give us a little more information, what is your setup? Versions of Kube and Kafka? How many Kafka &amp; ZK pods? How are you deploying your Kafka cluster (via a simple helm chart or an operator?) What are the exact symptoms you see when you upgrade your kube cluster? What errors do you get? What is the state of the Kafka cluster etc.? How do you monitor it?</p>
<p>But here are some points worth investigating.</p>
<ul>
<li>Are you spreading the Kafka/ZK pods correctly across the nodes/zones?</li>
<li>Do you set PDBs to a reasonable maxUnavailable setting?</li>
<li>What are your readiness/liveness probes for your Kafka/ZK pods?</li>
<li>Are your topics correctly replciated?</li>
</ul>
<p>I would strongly encourage you to use take a look at <a href="https://strimzi.io/" rel="nofollow noreferrer">https://strimzi.io/</a> which can be very helpful if you want to operate Kafka on Kube. It is open source operator and very well documented.</p>
