<h3>Situation</h3>
<ul>
<li>We have multiple Kubernetes pods of each deployment (= microservice)</li>
<li>Our application is massively Kafka-based</li>
<li>Normally each microservice consumes each Kafka topic in a single consumer group, which is good as we need that each Kafka events is consumed exactly once</li>
<li>Kubernetes launches and stops pods whenever it's needed</li>
</ul>
<h3>Problem</h3>
<ul>
<li>We have one topic, which must be consumed by all the running instances (pods) of a single microservice (to reset some internal status)</li>
</ul>
<h2>What we considered and why it doesn't work</h2>
<ul>
<li>Consume from the topic always in a new consumer group with a random name:<br>
NO, because as soon as a pod is killed, the consumer group remains for ever, with the constantly increasing lag. The number of detached consumer group without a listener grows over time.</li>
<li><a href="https://stackoverflow.com/q/69717216/2886891">Assigning a unique number in env inside each pod in a kubernetes cluster</a>:<br>
NO, because we use ReplicaSet and not StatefulSet</li>
</ul>
<h2>Requirements (summary)</h2>
<ul>
<li>All the events from one particular topic must be consumed by each running instance of the microservice, i.e. by each pod of one particular deployment.</li>
<li>Kubernetes uses ReplicaSet, the names of the pods have random a string appended.</li>
<li>No zombie consumer groups are allowed.</li>
</ul>
<h2>Question</h2>
<ul>
<li>How to solve it, either in Kafka or in Kubernetes or anywhere outside or around or in the middle of it?</li>
</ul>

Honza Zidek
<blockquote>
<p>the consumer group remains for ever</p>
</blockquote>
<p>If you don't commit the offsets, then no group is created.</p>
<blockquote>
<p>constantly increasing lag</p>
</blockquote>
<p>This is only an issue if you use <code>auto.offset.reset=earliest</code>.</p>
<blockquote>
<p>All the events from one particular topic must be consumed by each running instance of the microservice</p>
</blockquote>
<p>You can use <code>consumer.assign()</code> with the list of TopicPartition, which you can query with <code>AdminClient</code>, or hard-code.</p>
<p>Otherwise, you need to use a random group per pod since any static value will cause idle consumers when there are more pods than partitions.</p>
<p>Also, you are not required to use Kafka <code>__consumer_offsets</code> topic at  all. You can manually commit offset values in an external store, and use <code>assign</code> and <code>seek</code> functions. When you use an external store, then you can have better guarantees of exactly-once processing since you can additionally mark any offset as &quot;processed==true&quot;, and seek to the min(processed == false) offset before each seek.</p>
