<p>I have a kafka topic that has 5 partitions. I have 5 pods currently consuming these 5 partitions. However due to a particular need, I need each pod to only consume from its assigned partition. But due to pods all having the identical configuration on kubernetes I can't tell each pod which partition to consumer. What's the best way to achieve this? Should I just have 5 independent deployments instead?</p>
<p>That seems odd with all 5 having the same code. I've seen the use of stateful set, but that seems to rely on an index. Not sure how robust that approach is.</p>

casualprogrammer
<p>If you're able to change your deployment to a statefulset, each pod of your replica will contain a sequence number starting from 0. The podname is fixed like my-kafka-consumer-0, my-kafka-consumer-1, my-kafka-consumer-(N-1), and so on.</p>
<p>Next you should use the podname as an environment variable inside your pod which can be read by your application.</p>
<p>You can do it by set the env variable like this:</p>
<pre><code>env:
  - name: MY_POD_NAME
    valueFrom:
      fieldRef:
        fieldPath: metadata.name
</code></pre>
<p>If this is done, your application should be able to read this information from the environment variables. This is needed to identify a consumer inside the consumer group.</p>
<p>So the next thing what you're looking for it's called a static membership in your kafka client configuration.</p>
<p>You can reach this by setting the group.id and the group.instance.id</p>
<pre><code>'group.id': 'consumer-group'
'group.instance.id': '&lt;MY_POD_NAME FROM THE ENVIRONMENT VARS&gt;'
</code></pre>
<p>Not tested, but i think this should do the trick.</p>
