<p>I am using Microk8s and bitnami helm chart <a href="https://artifacthub.io/packages/helm/bitnami/mongodb" rel="nofollow noreferrer">here</a></p>
<p>I set up a replicaset of 3 replicas
mongo-0 (by definition this is Primary), mongo-1 and mongo-2</p>
<p>Bitnami makes the replicaset to always use mongo-0 (if available) as Primary replica. However the next can happen:  I find out I need to update the nodes, let's say to increase storage. To do this, I would need to:</p>
<ol>
<li>Drain the node running mongo-0. This automatically triggers a new election, and let's say mongo-1 is the new primary.</li>
<li>I will add to the cluster a new node (with more capacity).</li>
</ol>
<p>This will make the mongodb replicaset to assign a mongo-0 pod to the new node. However, the new node is empty, so the persistent volume where I store the database (lets say /mnt/mongo) is empty.</p>
<p>I would expect that the current primary replica will finish populating the database to the new replica (mongo-0, and therefore its Persistent Volume) and ONLY when that is done, then make mongo-0 the primary.</p>
<p>However I saw that mongo-0 becomse primary without any data being copied to it from the previous primary, effectively deleting the whole database, since now the primary node states that the database is empty.</p>
<p>How is that possible? What am I missing here?</p>

FCR
<p>I am not familiar with your exact management tools but the scaling process you described is wrong. You should not be removing 1 out of 3 nodes out of the replica set at any point, at least not in a production environment.</p>
<p>To replace a RS node:</p>
<ol>
<li>Add a fourth node with desired parameters.</li>
<li>Set node priorities such that the node you want to remove has a lower priority than any of the other nodes.</li>
<li>Wait for the newly added node to have an acceptable <a href="https://bluemedora.com/troubleshoot-mongodb-replication-lag/" rel="nofollow noreferrer">replication lag</a>.</li>
<li>Ensure the primary is not the node you want to remove.</li>
<li>Remove the node you want to remove from the RS.</li>
</ol>
<p>Expecting generic software to automatically figure out when #3 completes and move on to #4 correctly is, I would say, rather optimistic. Maybe MongoDB ops manager would do that.</p>
<p>Your post contains a number of other incorrect statements about how MongoDB operates. For example, a node that has no data cannot become a primary in a replica set with data. Perhaps you have other configuration issues going on and you actually have independent nodes/multiple replica sets in what you think is a single deployment.</p>
