title: Kafka - Troubleshooting .NotEnoughReplicasException
body: <p>I started seeing the following error </p>
  
  <pre><code>[2020-06-12 20:09:01,324] ERROR [ReplicaManager broker=3] Error processing append operation on partition __consumer_offsets-10 (kafka.server.ReplicaManager)
  org.apache.kafka.common.errors.NotEnoughReplicasException: The size of the current ISR Set(3) is insufficient to satisfy the min.isr requirement of 2 for partition __consumer_offsets-10    
  </code></pre>
  
  <blockquote>
    <p>My setup is having three brokers and all brokers are up. Couple of
    things i did before this error was about pop up</p>
    
    <p>I configured min.isr to be 2 in all the brokers. I created a topic
    with replication factor 3 and starting producing the message from a
    producer with ack = 1 with two brokers down. I brought up all the
    brokers and started consumer.</p>
  </blockquote>
  
  <ol>
  <li>How to go about troubleshooting this error      </li>
  <li>Consumer is also NOT be able to see this message ( not sure why, the message is supposed to be treated as "committed" as one broker was up when the producer was running)</li>
  </ol>
  
  <p><strong>Couple of facts</strong></p>
  
  <p>It is interesting to see rebalancing didnt happen <em>yet</em> WRT preferred leader starategy</p>
  
  <pre><code>$ kafka-topics --zookeeper 127.0.0.1:2181 --topic stock-prices --describe                             
    Topic: stock-prices     PartitionCount: 3       ReplicationFactor: 3    Configs: min.insync.replicas=2
    Topic: stock-prices     Partition: 0    Leader: 1       Replicas: 1,3,2 Isr: 1,2,3            
    Topic: stock-prices     Partition: 1    Leader: 1       Replicas: 2,1,3 Isr: 1,2,3            
    Topic: stock-prices     Partition: 2    Leader: 1       Replicas: 3,2,1 Isr: 1,2,3
  </code></pre>