<p>I have a k8s cluster that runs just fine. It has several standalone mongodb statefulsets connected via NFC. The problem is, whenever their is a power outage, the mongodb databases get corrupt:</p>
<pre><code>{&quot;t&quot;:{&quot;$date&quot;:&quot;2021-10-15T13:10:06.446+00:00&quot;},&quot;s&quot;:&quot;W&quot;,  &quot;c&quot;:&quot;STORAGE&quot;,  &quot;id&quot;:22271,   &quot;ctx&quot;:&quot;initandlisten&quot;,&quot;msg&quot;:&quot;Detected unclean shutdown - Lock file is not empty&quot;,&quot;attr&quot;:{&quot;lockFile&quot;:&quot;/data/db/mongod.lock&quot;}}

{&quot;t&quot;:{&quot;$date&quot;:&quot;2021-10-15T13:10:07.182+00:00&quot;},&quot;s&quot;:&quot;E&quot;,  &quot;c&quot;:&quot;STORAGE&quot;,  &quot;id&quot;:22435,   &quot;ctx&quot;:&quot;initandlisten&quot;,&quot;msg&quot;:&quot;WiredTiger error&quot;,&quot;attr&quot;:{&quot;error&quot;:0,&quot;message&quot;:&quot;[1634303407:182673][1:0x7f9515eb7a80], file:WiredTiger.wt, connection: __wt_block_read_off, 283: WiredTiger.wt: read checksum error for 4096B block at offset 12288: block header checksum of 0xc663f362 doesn't match expected checksum of 0xb8e27418&quot;}}
</code></pre>
<p>The pods status remain at CrashLoopBackOff so I cannot do <code>kubectl exec -it usersdb-0 -- mongod --repair</code> because it is not running.</p>
<p>I have tried deleting wiredTiger.lock and mongod.lock but nothing seems to work. How can I repair this databases?</p>

<p>well after several attempts I think I have finally made some breakthrough so I wanted to leave this here for someone else.</p>
<p>Since the mongodb is not running, add the command</p>
<pre><code>command: [&quot;sleep&quot;]
args: [&quot;infinity&quot;]
</code></pre>
<p>in the resource file (hoping it is a statefulset).
Then repair the database using the command</p>
<pre><code>kubectl exec -it &lt;NAME-OF-MONGODB-POD&gt; -- mongod --dbpath /data/db --repair
</code></pre>
<p>This will repair the standalone mongodb pod. Now remove the comment, apply the resource yaml file then kill the pod to recreate it afresh.</p>
<p>Now the mongodb pod should be working fine.</p>
