<p>I have an app that get data from a third-party data source, it will send data to my app automatically and I can't filter it, I can only receive all. When data arrive, my app will transmit this data to a rocketmq topic.</p>
<p>Now I have to make this app a container and deploy it in k8s deployment with 3 replica. But these pods will all get same data and send to the same rocketmq topic.</p>
<p>How do I make this app horizontal scalable without sending duplicate msg to the same rocketmq topic?</p>

Wayne Chang
<blockquote>
<p>Now I have to make this app a container and deploy it in k8s deployment with 3 replica. But these pods will all get same data and send to the same rocketmq topic.</p>
</blockquote>
<blockquote>
<p>There is no request. My app connect to a server and it will send data to app by TCP. Every Pod will connect to that server.</p>
</blockquote>
<p>If you want to do this with more than one instance, they need to coordinate in some way.</p>
<p><a href="https://medium.com/hybrid-cloud-hobbyist/leader-election-architecture-kubernetes-32600da81e3c" rel="nofollow noreferrer">Leader Election pattern</a> is a way to run multiple instances, but only one can be active (e.g. when you read from the same queue). This is a pattern to coordinate - only one instance is active at the time. So this pattern only use your replicas for <em>higher availability</em>.</p>
<p>If you want that all your replicas actively work, this can be done with techniques like <a href="https://stackoverflow.com/a/58678622/213269">sharding or partitioning</a>. This is also how e.g. <a href="https://www.instaclustr.com/the-power-of-kafka-partitions-how-to-get-the-most-out-of-your-kafka-cluster/" rel="nofollow noreferrer">Kafka</a> (e.g. similar to a queue) makes <strong>concurrent work</strong> from queues.</p>
<p>There are other ways to solve this problem as well, e.g. to implement some form of locks to coordinate - but partitioning or sharding as in Kafka is probably the most &quot;cloud native&quot; solution.</p>
