title: MongoDB rollback and replica set issue
body: <p>We have a three member MongoDB replica set. One of the nodes is trying to do a rollback and fails. As a result this Mongo POD in Crashloop. What is &quot;Stable Timestamp&quot; in MongoDB ? What is &quot;top&quot; of Oplog?</p>
  <pre><code>initiate system update and upgrade stuck in mongodb
  2021-04-08T06:51:06.690+0000 I STORAGE [initandlisten] WiredTiger record store oplog processing took 554ms
  2021-04-08T06:51:06.693+0000 I STORAGE [initandlisten] Timestamp monitor starting
  2021-04-08T06:51:06.697+0000 I CONTROL [initandlisten]
  2021-04-08T06:51:06.697+0000 I CONTROL [initandlisten] ** WARNING: You are running on a NUMA machine.
  2021-04-08T06:51:06.697+0000 I CONTROL [initandlisten] ** We suggest launching mongod like this to avoid performance problems:
  2021-04-08T06:51:06.697+0000 I CONTROL [initandlisten] ** numactl --interleave=all mongod [other options]
  2021-04-08T06:51:06.698+0000 I CONTROL [initandlisten]
  2021-04-08T06:51:06.718+0000 I SHARDING [initandlisten] Marking collection local.system.replset as collection version: &lt;unsharded&gt;
  2021-04-08T06:51:06.720+0000 I STORAGE [initandlisten] Flow Control is enabled on this deployment.
  2021-04-08T06:51:06.720+0000 I SHARDING [initandlisten] Marking collection admin.system.roles as collection version: &lt;unsharded&gt;
  2021-04-08T06:51:06.720+0000 I SHARDING [initandlisten] Marking collection admin.system.version as collection version: &lt;unsharded&gt;
  2021-04-08T06:51:06.722+0000 I SHARDING [initandlisten] Marking collection local.startup_log as collection version: &lt;unsharded&gt;
  2021-04-08T06:51:06.722+0000 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/var/data/mongodb/diagnostic.data'
  2021-04-08T06:51:06.723+0000 I SHARDING [initandlisten] Marking collection local.replset.minvalid as collection version: &lt;unsharded&gt;
  2021-04-08T06:51:06.723+0000 I SHARDING [initandlisten] Marking collection local.replset.election as collection version: &lt;unsharded&gt;
  2021-04-08T06:51:06.725+0000 I REPL [initandlisten] Rollback ID is 2
  2021-04-08T06:51:06.726+0000 I REPL [initandlisten] Recovering from stable timestamp: Timestamp(1617686958, 1) (top of oplog:
  { ts: Timestamp(1617686416, 1), t: 32 }
  , appliedThrough:
  { ts: Timestamp(1617686958, 1), t: 36 }
  , TruncateAfter: Timestamp(0, 0))
  2021-04-08T06:51:06.726+0000 I REPL [initandlisten] Starting recovery oplog application at the stable timestamp: Timestamp(1617686958, 1)
  2021-04-08T06:51:06.726+0000 F REPL [initandlisten] Applied op { : Timestamp(1617686958, 1) } not found. Top of oplog is { : Timestamp(1617686416, 1) }.
  2021-04-08T06:51:06.726+0000 F - [initandlisten] Fatal Assertion 40313 at src/mongo/db/repl/replication_recovery.cpp 511
  2021-04-08T06:51:06.726+0000 F - [initandlisten] \n\n***aborting after fassert() failure\n\n
  </code></pre>
  <p>Update:
  Precisely I am trying to understand, what is the stable timestamp being referred to here. why would top of oplog matter?</p>
  <p>Recovering from stable timestamp: Timestamp(1617686958, 1) (top of oplog:
  { ts: Timestamp(1617686416, 1), t: 32 }
  , appliedThrough:
  { ts: Timestamp(1617686958, 1), t: 36 }
  , TruncateAfter: Timestamp(0, 0))</p>
  <p>Why does the assertion fail here?</p>
  <pre><code>2021-04-08T06:51:06.726+0000 F REPL [initandlisten] Applied op { : Timestamp(1617686958, 1) } not found. Top of oplog is { : Timestamp(1617686416, 1) }.
  2021-04-08T06:51:06.726+0000 F - [initandlisten] Fatal Assertion 40313 at src/mongo/db/repl/replication_recovery.cpp 511
  </code></pre>