title: Kafka Streams: Kafka Streams application stuck rebalancing
body: <p>After all Kafka brokers restart to upgrade offset.retention.minutes setting (to increase it to 60 days), the Kafka Streams application consuming there were stuck, and the consumer group shows rebalancing: </p>
  
  <pre><code>bin/kafka-consumer-groups.sh --bootstrap-server ${BOOTSTRAP_SERVERS} --describe --group stream-processor | sort 
  
  Warning: Consumer group 'stream-processor' is rebalancing.
  
  TOPIC                  PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID     HOST            CLIENT-ID
  customers              0          84263           84288           25              -               -               -
  customers              1          71731           85068           13337           -               -               -
  customers              10         71841           84801           12960           -               -               -
  customers              11         84273           84336           63              -               -               -
  customers              12         84189           84297           108             -               -               -
  customers              13         83969           84051           82              -               -               -
  customers              14         84693           84767           74              -               -               -
  customers              15         84472           84556           84              -               -               -
  customers              2          84175           84239           64              -               -               -
  customers              3          71719           71719           0               -               -               -
  customers              4          71446           84499           13053           -               -               -
  customers              5          84291           84361           70              -               -               -
  customers              6          71700           71700           0               -               -               -
  customers              7          72003           85235           13232           -               -               -
  customers              8          84521           84587           66              -               -               -
  customers              9          71513           71513           0               -               -               -
  customers-intermediate 0          102774          102792          18              -               -               -
  customers-intermediate 1          102931          103028          97              -               -               -
  customers-intermediate 10         102883          102965          82              -               -               -
  customers-intermediate 11         102723          102861          138             -               -               -
  customers-intermediate 12         102797          102931          134             -               -               -
  customers-intermediate 13         102339          102460          121             -               -               -
  customers-intermediate 14         103227          103321          94              -               -               -
  customers-intermediate 15         103231          103366          135             -               -               -
  customers-intermediate 2          102637          102723          86              -               -               -
  customers-intermediate 3          84620           103297          18677           -               -               -
  customers-intermediate 4          102596          102687          91              -               -               -
  customers-intermediate 5          102980          103071          91              -               -               -
  customers-intermediate 6          84385           103058          18673           -               -               -
  customers-intermediate 7          103559          103652          93              -               -               -
  customers-intermediate 8          103243          103312          69              -               -               -
  customers-intermediate 9          84211           102772          18561           -               -               -
  events                 15         11958555        15231834        3273279         -               -               -
  events                 3          1393386         16534651        15141265        -               -               -
  events                 4          1149540         15390069        14240529        -               -               -
  visitors               15         2774874         2778873         3999            -               -               -
  visitors               3          603242          603324          82              -               -               -
  visitors               4          565266          565834          568   
  </code></pre>
  
  <p>The streaming application was restarted too, and afterwards I could see some processing logs for about 20 hours and then stopped processing.</p>
  
  <p>It's been like this for two days. But it is also worth mentioning that all topics you see above have 16 partitions, but some show three of them (visitors, events). However I can describe the topics and they have their partitions well distributed as usual and I can find nothing strange there. </p>
  
  <p>What could have happened? 
  After application restart, I can see all partitions again, and the applications consuming from topic partitions. However many (most) partitions had lost their offsets. Since I changed the <code>offset.retention.minutes</code> setting, this should not have happened.</p>
  
  <pre><code>events                 0          -               14538261        -               stream-processor-01f7ecea-4e50-4505-a8e7-8c536058b7bc-StreamThread-1-consumer-a8a6e989-d6c1-472f-aec5-3ae637d87b9e /ip2  stream-processor-01f7ecea-4e50-4505-a8e7-8c536058b7bc-StreamThread-1-consumer
  events                 1          49070           13276094        13227024        stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer-69da45eb-b0b2-4ad8-a831-8aecf6849892 /ip1  stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer
  events                 10         -               15593746        -               stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer-69da45eb-b0b2-4ad8-a831-8aecf6849892 /ip1  stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer
  events                 11         -               15525487        -               stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer-69da45eb-b0b2-4ad8-a831-8aecf6849892 /ip1  stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer
  events                 12         -               21863908        -               stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer-69da45eb-b0b2-4ad8-a831-8aecf6849892 /ip1  stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer
  events                 13         -               15810925        -               stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer-69da45eb-b0b2-4ad8-a831-8aecf6849892 /ip1  stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer
  events                 14         -               13509742        -               stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer-69da45eb-b0b2-4ad8-a831-8aecf6849892 /ip1  stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer
  events                 15         11958555        15231834        3273279         stream-processor-01f7ecea-4e50-4505-a8e7-8c536058b7bc-StreamThread-1-consumer-a8a6e989-d6c1-472f-aec5-3ae637d87b9e /ip2  stream-processor-01f7ecea-4e50-4505-a8e7-8c536058b7bc-StreamThread-1-consumer
  ...
  </code></pre>
  
  <p>Kafka 1.1.0, Kafka Streams 1.0</p>
  
  <p><strong>UPDATE</strong> Still happening on 2.1.1.</p>