<p>I have a Kafka cluster running on Kubernetes (on AWS). Each broker has a corresponding external loadbalancer (ELB) and afaict, Kafka's <code>advertised.listeners</code> have been set appropriately so that the ELB's DNS names get returned when clients query for broker information. Most of the setup is similar to the one mentioned <a href="https://github.com/Yolean/kubernetes-kafka/" rel="nofollow noreferrer">here</a>.</p>

<p>I created a kafka consumer without specifying any group-id. With this consumer, reading messages from a topic worked just fine. However, if I set a group-id when creating the kafka consumer, I get back the following error messages:</p>

<pre><code>2018-01-30    22:04:16,763.763.313055038:kafka.cluster:140735643595584:INFO:74479:Group coordinator for my-group-id is BrokerMetadata(nodeId=2, host=u'a17ee9a8a032411e8a3c902beb474154-867008169.us-west-2.elb.amazonaws.com', port=32402, rack=None)
2018-01-30 22:04:16,763.763.804912567:kafka.coordinator:140735643595584:INFO:74479:Discovered coordinator 2 for group my-group-id
2018-01-30 22:04:16,764.764.270067215:kafka.coordinator.consumer:140735643595584:INFO:74479:Revoking previously assigned partitions set([]) for group my-group-id
2018-01-30 22:04:16,866.866.26291275:kafka.coordinator:140735643595584:INFO:74479:(Re-)joining group my-group-id
2018-01-30 22:04:16,898.898.787975311:kafka.coordinator:140735643595584:INFO:74479:Joined group 'my-group-id' (generation 1) with member_id kafka-python-1.3.5-e31607c2-45ec-4461-8691-260bb84c76ba
2018-01-30 22:04:16,899.899.425029755:kafka.coordinator:140735643595584:INFO:74479:Elected group leader -- performing partition assignments using range
2018-01-30 22:04:16,936.936.614990234:kafka.coordinator:140735643595584:WARNING:74479:Marking the coordinator dead (node 2) for group my-group-id: [Error 15] GroupCoordinatorNotAvailableError.
2018-01-30 22:04:17,069.69.8890686035:kafka.cluster:140735643595584:INFO:74479:Group coordinator for my-group-id is BrokerMetadata(nodeId=2, host=u'my-elb.us-west-2.elb.amazonaws.com', port=32402, rack=None)
</code></pre>

<p><code>my-elb.us-west-2.elb.amazonaws.com:32402</code> is accessible from the client. I used <code>kafkacat</code> and set <code>my-elb.us-west-2.elb.amazonaws.com:32402</code> as the broker address, it was able to list topics, consume topics, etc.</p>

<p>Any ideas what might be wrong?</p>

<p>Marking the coordinator dead happens when there is a Network communication error between the Consumer Client and the Coordinator (Also this can happen when the Coordinator dies and the group needs to rebalance). There are a variety of situations (offset commit request, fetch offset, etc) that can cause this issue. So to find the root cause issue you need to set the logging level to trace and debug :</p>

<blockquote>
  <p>logging.level.org.apache.kafka=TRACE</p>
</blockquote>
