<p>We are trying to update a document using <code>updateOne()</code>. The filter identifies the document and we update an attribute of the document using <code>$set</code>. This update job is triggered by a cron job every minute.</p>
<p>Say, original document is <code>{_id: &quot;USER1001&quot;, status: &quot;INACTIVE&quot;}</code>. We update it by calling <code>updateOne()</code> using the filter <code>{_id: &quot;USER1001&quot;, status: &quot;INACTIVE&quot;}</code> and updating the status field as <code>{&quot;$set&quot;:{status:&quot;ACTIVE&quot;}}</code>. We look at resultant value of <code>modifiedCount</code> and expect it to be 1, to claim that the <code>updateOne()</code> operation was successful. This then triggers downstream jobs.</p>
<p>The application is running in Kubernetes and we have scaled it. When we test the system under load, two simultaneous <code>updateOne()</code> calls are made to the same document with the same filter, from two different pods and both returns <code>modifiedCount</code> as 1.</p>
<p>We expect <code>modifiedCount</code> to be 1 for one for the pod and 0 for the other. But for some documents we see this result.</p>
<p><em>Sample code for reference</em></p>
<pre><code>// cron job that calls update() every minute

func update(){
  filter := bson.D{{&quot;_id&quot;, &quot;USER1001&quot;}, {&quot;status&quot;, &quot;INACTIVE&quot;}}
  result, err := collection.UpdateOne(context.Background(), filter, bson.M{&quot;$set&quot;: bson.M{&quot;status&quot;: &quot;ACTIVE&quot;}})
  
  if result.ModifiedCount != 1 {
    // no update done
  } else {
    // call downstream jobs
  }
}
</code></pre>
<p><em>Sample log lines that we have from the application pods</em></p>
<ul>
<li>POD-1</li>
</ul>
<blockquote>
<p>[2020-11-20 17:30:58.610518875 +0000 UTC] [DEBUG] [myJob-7dc8b78bcf-c4677] update() :: Update result :: USER1001 / Matched=1 / Modified=1</p>
</blockquote>
<ul>
<li>POD-2</li>
</ul>
<blockquote>
<p>[2020-11-20 17:30:58.409843674 +0000 UTC] [DEBUG] [myJob-7dc8b78bcf-jd7m8] update() :: Update result :: USER1001 / Matched=1 / Modified=1</p>
</blockquote>
<p><strong>Question here is -</strong></p>
<ol>
<li>Has any one else seen this behaviour?</li>
<li>What could be causing this issue?</li>
</ol>
<p><strong>Additional info,</strong></p>
<ul>
<li>The application is in Go</li>
<li>Mongo is 4.4 and</li>
<li>We are using the latest mongo-driver for go.</li>
<li>The cron job runs every minute, not on the stroke of a minute. Which means that,
<ul>
<li>POD-1 might run it at,
<ul>
<li>10:00:23</li>
<li>10:01:23</li>
<li>10:02:23 etc</li>
</ul>
</li>
<li>POD-2 might run it at,
<ul>
<li>10:00:35</li>
<li>10:01:35</li>
<li>10:02:35 etc</li>
</ul>
</li>
</ul>
</li>
</ul>

jerrymannel
<p>If you read this document first, find out that it is inactive and then decide to active it, this is a common race condition you have to deal with. Another process does the same thing, and then both update the document.</p>
<p>There are ways to prevent this. Mongodb document level operations are atomic, so the simplest solution for your case is to change your filter to <code>{_id:&quot;USER1001&quot;,&quot;status&quot;:&quot;INACTIVE&quot;}</code> to make sure the document is inactive the moment you update it. Then only one node will successfully update the document though multiple nodes might attempt to update it.</p>
