title: Kafka Consumer Heartbeat Thread Block on Java Nio
body: <p>Software libraries’ Versions in use: Spring 2.2.3.Release, Spring kafka 2.x , Kafka-Client 2.0.0
  Java Version: OpenJdk 1.8
  Platform Kafka version:  Apache kafka 2.2.0 (<a href="https://docs.confluent.io/5.2.1/release-notes.html#apache-kafka-2-2-0-cp2" rel="nofollow noreferrer">https://docs.confluent.io/5.2.1/release-notes.html#apache-kafka-2-2-0-cp2</a>)</p>
  
  <p>Configuration: Topic has 6 partitions.
  Data rate: The incoming data rate is 3-4 msgs/second. </p>
  
  <p>Scenario:
  With the above configuration a test was run continuously for 3 days. After 2.5 days, I encountered problem in terms of not receiving messages from topic in our consumer group.
  Upon detailed investigation, I found out that consumer threads were blocked precisely 18 threads were in blocked state. Also, the thread graph is suggesting that consumer threads were waiting on “Java.nio.</p>
  
  <p>PFA logs and screenshots .
  a.) Logs
  b.) Screenshots</p>
  
  <p>a)Logs</p>
  
  <pre><code>
  
  "kafka-coordinator-heartbeat-thread | release-registry-group" #128 daemon prio=5 os_prio=0 tid=0x00007f1954001800 nid=0x8a waiting for monitor entry [0x00007f197f7f6000]
     java.lang.Thread.State: BLOCKED (on object monitor)
          at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:1005)
          - waiting to lock &lt;0x00000000c3382070&gt; (a org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
  
  "org.springframework.kafka.KafkaListenerEndpointContainer#1-2-C-1" #127 prio=5 os_prio=0 tid=0x00007f1b6dbc1800 nid=0x89 runnable [0x00007f197f8f6000]
     java.lang.Thread.State: RUNNABLE
          at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
          at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
          at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
          at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
          - locked &lt;0x00000000c33822d8&gt; (a sun.nio.ch.Util$3)
          - locked &lt;0x00000000c33822c8&gt; (a java.util.Collections$UnmodifiableSet)
          - locked &lt;0x00000000c33822e8&gt; (a sun.nio.ch.EPollSelectorImpl)
          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
          at org.apache.kafka.common.network.Selector.select(Selector.java:689)
          at org.apache.kafka.common.network.Selector.poll(Selector.java:409)
          at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:510)
          at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:271)
          at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:242)
          at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:218)
          at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:230)
          - locked &lt;0x00000000c3382070&gt; (a org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
          at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:314)
          at org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1218)
          at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1175)
          at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1154)
          at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:732)
          at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:689)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          at java.lang.Thread.run(Thread.java:748)
  
  
  </code></pre>
  
  <p>b.) Screenshots
  <a href="https://i.stack.imgur.com/vDdS6.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/vDdS6.png" alt="enter image description here"></a></p>
  
  <p><a href="https://i.stack.imgur.com/P926h.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/P926h.png" alt="enter image description here"></a></p>
  
  <p><a href="https://i.stack.imgur.com/7BQEo.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/7BQEo.png" alt="enter image description here"></a></p>
  
  <p>I would like to understand root cause and possible solutions to fix it and thanks in advance. </p>