<p><strong>My Application:</strong></p>
<p>I have one production application named <strong>&quot;Realtime Student Streaming Application&quot;</strong>.
and it is connected to MongoDB1 and i have Student table(collection) inside of it.</p>
<p><strong>My Task:</strong></p>
<ol>
<li>My Application will Listen for any insertion happened in student table.</li>
<li>Once insertion happened, mongo will give the inserted record to my listener class.</li>
<li>Once the record came to java class I will insert it into new database called MongoDB2.</li>
</ol>
<p><strong>What I did:</strong></p>
<ol>
<li>I deployed my application in OpenShift cluster and it has 5 pods running on it.</li>
<li>On this case, If any insertion is happened in MongoDB1, I'm receiving inserted data in all my 5 pods.</li>
</ol>
<p><strong>Current Output:</strong></p>
<p>Pod 1: I will process inserted-document 1,2,3,4,5</p>
<p>Pod 2: I will process inserted-document 1,2,3,4,5</p>
<p>Pod 3: I will process inserted-document 1,2,3,4,5</p>
<p>Pod 4: I will process inserted-document 1,2,3,4,5</p>
<p>Pod 5: I will process inserted-document 1,2,3,4,5</p>
<p><strong>Expected Output:</strong></p>
<p>Pod 1: I need to process only inserted-document 1</p>
<p>Pod 2: I need to process only inserted-document 2</p>
<p>Pod 3: I need to process only inserted-document 3</p>
<p>Pod 4: I need to process only inserted-document 4</p>
<p>Pod 5: I need to process only inserted-document 5</p>
<p><strong>Q1: Let say if i'm inserting 5 documents, How it will be shared between 5 pods.</strong></p>
<p><strong>Q2: In General, while using Kubernetes/Openshift , How to equally split data between multiple pods?</strong></p>

Vaishagkumar techy
<p>This really isn't an OpenShift or Kubernetes question, it's an application design question. Fundamentally, it's up to <em>you</em> to shard the data in some fashion. Fundamentally it's a difficult problem to solve, and even tougher to solve once you start tackling problems around transactionality, scaling up/down and preserving message order.</p>
<p>Therefore, one common way to do this is to use Kafka. Either manually, or with something like Debezium, stream the changes from the DB into a Kafka topic. You can then use Kafka partitions and consumer groups to automatically divide the work between a dynamic number of workers. This may seem like overkill, but it solves a lot of the problems I mentioned above.</p>
<p>Of course you can also try to do this manually, just manually sharding the work and having each worker use Mongo features like findAndModify to &quot;lock&quot; specific updates and then update them again once completed. But then you have to build a lot of those features yourself, including how to recover if a worker locks an update but then fails before processing the change.</p>
