title: Kafka Cluster on EC2 facing "Shrinking ISR for partition" and "Cached zkVersion [] not equal to that in zookeeper, skip updating ISR"
body: <p>I have a 3 node kafka cluster built on <code>t2.medium</code> instances. Zookeeper and Broker are deployed in same ec2 instance. ec2 instances are spread across 3 diff av zones within a region.
  The Setup is as following:</p>
  
  <ol>
  <li>Zookeeper has following memory settings  : <code>export KAFKA_HEAP_OPTS="-Xmx512M -Xms512M"</code></li>
  <li>Broker has following memory settings     : <code>export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"</code></li>
  <li><code>Kafka Version - 0.10.1.1</code> which is officially compliant with <code>spring-boot 1.5.10</code></li>
  <li>Zookeeper bundled with Kafka is being used for setup.</li>
  </ol>
  
  <p>Traffic Pattern: The traffic is not high (may be 4 msgs/sec), however huge amount of messages (500 msgs/sec) can arrive in short bursts of time [2-3 mins].</p>
  
  <p>Issues Facing:
  1. The following gets reported in broker server.log</p>
  
  <p><code>INFO Partition [topic1,0] on broker 0: Shrinking ISR for partition [topic1,0] from 0,1,2 to 0 (kafka.cluster.Partition)</code></p>
  
  <p><code>INFO Partition [topic1,0] on broker 0: Cached zkVersion [8] not equal to that in zookeeper, skip updating ISR (kafka.cluster.Partition)</code></p>
  
  <p>This makes the cluster unstable and until a rolling restart of all brokers (also zookeepers sometime) is performed, the cluster never recovers on it's own.</p>
  
  <p>There is no noticeable nw issue when checked from Metrics available in AWS EC2 Console, except spike in traffic. Also nothing in logs from nw perspective, when the issue starts.
  It looks like there is a GC run, at the time when it starts reporting and then continuously keeps reporting on <code>kafkaServer-GC.log</code>, until rolling restart of the whole cluster is performed.</p>
  
  <ol>
  <li>What should be the correct instance type to handle this kind of scenario? What would be the ideal memory settings advisable? Is there any other configuration that can be corrected in this regard?</li>
  <li>Is there a way to find out the problematic node? </li>
  <li>How do we decide whether the broker restart alone can help, or all brokers needs to be restarted as well? </li>
  <li>How do we decide whether the zookeeper restart also is needed, and/or all zookeeper instances needs restart as well? Is there a way to use 4 letter words with the <code>zookeeper-shell.sh</code> comes bundled with kafka?</li>
  <li>What can be a proactive monitoring step to detect this occurrence early?</li>
  </ol>
  
  <p>Some guidance is badly needed here and needless to mention it's Extremely Appreciated!</p>