<p>I'm running a kubernetes cluster (bare metal) with a mongodb (version 4, as my server cannot handle newer versions) replicaset (2 replicas), which is initially working, but from time to time (sometimes 24 hours, somtimes 10 days) one or more mongodb pods are failing.</p>
<pre><code>Warning  BackOff  2m9s (x43454 over 6d13h)  kubelet  Back-off restarting failed container
</code></pre>
<p>The relevant part of the logs should be</p>
<pre><code>DBPathInUse: Unable to create/open the lock file: /bitnami/mongodb/data/db/mongod.lock (Read-only file system). Ensure the user executing mongod is the owner of the lock file and has the appropriate permissions. Also make sure that another mongod instance is not already running on the /bitnami/mongodb/data/db directory
</code></pre>
<p>But I do not change anything and initially it is working. Also the second pod is currently running (but which will fail the next days).</p>
<p>I'm using longhorn (before I tried nfs) for the storage and I installed mongodb using bitnami helm chart with these values:</p>
<pre><code>image:
  registry: docker.io
  repository: bitnami/mongodb
  digest: &quot;sha256:916202d7af766dd88c2fff63bf711162c9d708ac7a3ffccd2aa812e3f03ae209&quot; # tag: 4.4.15
  pullPolicy: IfNotPresent
architecture: replicaset
replicaCount: 2
updateStrategy:
  type: RollingUpdate
containerPorts:
  mongodb: 27017
auth:
  enabled: true
  rootUser: root
  rootPassword: &quot;password&quot;
  usernames: [&quot;user&quot;]
  passwords: [&quot;userpass&quot;]
  databases: [&quot;db&quot;]

service:
  portName: mongodb
  ports:
    mongodb: 27017

persistence:
  enabled: true
  accessModes:
    - ReadWriteOnce
  size: 8Gi

volumePermissions:
  enabled: true

livenessProbe:
  enabled: false
readinessProbe:
  enabled: false
</code></pre>
<p><strong>logs</strong></p>
<pre><code>mongodb 21:25:05.55 INFO  ==&gt; Advertised Hostname: mongodb-1.mongodb-headless.mongodb.svc.cluster.local
mongodb 21:25:05.55 INFO  ==&gt; Advertised Port: 27017
mongodb 21:25:05.56 INFO  ==&gt; Pod name doesn't match initial primary pod name, configuring node as a secondary
mongodb 21:25:05.59 
mongodb 21:25:05.59 Welcome to the Bitnami mongodb container
mongodb 21:25:05.60 Subscribe to project updates by watching https://github.com/bitnami/containers
mongodb 21:25:05.60 Submit issues and feature requests at https://github.com/bitnami/containers/issues
mongodb 21:25:05.60 
mongodb 21:25:05.60 INFO  ==&gt; ** Starting MongoDB setup **
mongodb 21:25:05.64 INFO  ==&gt; Validating settings in MONGODB_* env vars...
mongodb 21:25:05.78 INFO  ==&gt; Initializing MongoDB...
mongodb 21:25:05.82 INFO  ==&gt; Deploying MongoDB with persisted data...
mongodb 21:25:05.83 INFO  ==&gt; Writing keyfile for replica set authentication...

mongodb 21:25:05.88 INFO  ==&gt; ** MongoDB setup finished! **
mongodb 21:25:05.92 INFO  ==&gt; ** Starting MongoDB **

{&quot;t&quot;:{&quot;$date&quot;:&quot;2022-10-29T21:25:05.961+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;CONTROL&quot;,  &quot;id&quot;:20698,   &quot;ctx&quot;:&quot;main&quot;,&quot;msg&quot;:&quot;***** SERVER RESTARTED *****&quot;}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2022-10-29T21:25:05.963+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;CONTROL&quot;,  &quot;id&quot;:23285,   &quot;ctx&quot;:&quot;main&quot;,&quot;msg&quot;:&quot;Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'&quot;}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2022-10-29T21:25:05.968+00:00&quot;},&quot;s&quot;:&quot;W&quot;,  &quot;c&quot;:&quot;ASIO&quot;,     &quot;id&quot;:22601,   &quot;ctx&quot;:&quot;main&quot;,&quot;msg&quot;:&quot;No TransportLayer configured during NetworkInterface startup&quot;}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2022-10-29T21:25:05.968+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;NETWORK&quot;,  &quot;id&quot;:4648601, &quot;ctx&quot;:&quot;main&quot;,&quot;msg&quot;:&quot;Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize.&quot;}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2022-10-29T21:25:05.969+00:00&quot;},&quot;s&quot;:&quot;W&quot;,  &quot;c&quot;:&quot;ASIO&quot;,     &quot;id&quot;:22601,   &quot;ctx&quot;:&quot;main&quot;,&quot;msg&quot;:&quot;No TransportLayer configured during NetworkInterface startup&quot;}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2022-10-29T21:25:06.011+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;STORAGE&quot;,  &quot;id&quot;:4615611, &quot;ctx&quot;:&quot;initandlisten&quot;,&quot;msg&quot;:&quot;MongoDB starting&quot;,&quot;attr&quot;:{&quot;pid&quot;:1,&quot;port&quot;:27017,&quot;dbPath&quot;:&quot;/bitnami/mongodb/data/db&quot;,&quot;architecture&quot;:&quot;64-bit&quot;,&quot;host&quot;:&quot;mongodb-1&quot;}}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2022-10-29T21:25:06.011+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;CONTROL&quot;,  &quot;id&quot;:23403,   &quot;ctx&quot;:&quot;initandlisten&quot;,&quot;msg&quot;:&quot;Build Info&quot;,&quot;attr&quot;:{&quot;buildInfo&quot;:{&quot;version&quot;:&quot;4.4.15&quot;,&quot;gitVersion&quot;:&quot;bc17cf2c788c5dda2801a090ea79da5ff7d5fac9&quot;,&quot;openSSLVersion&quot;:&quot;OpenSSL 1.1.1n  15 Mar 2022&quot;,&quot;modules&quot;:[],&quot;allocator&quot;:&quot;tcmalloc&quot;,&quot;environment&quot;:{&quot;distmod&quot;:&quot;debian10&quot;,&quot;distarch&quot;:&quot;x86_64&quot;,&quot;target_arch&quot;:&quot;x86_64&quot;}}}}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2022-10-29T21:25:06.012+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;CONTROL&quot;,  &quot;id&quot;:51765,   &quot;ctx&quot;:&quot;initandlisten&quot;,&quot;msg&quot;:&quot;Operating System&quot;,&quot;attr&quot;:{&quot;os&quot;:{&quot;name&quot;:&quot;PRETTY_NAME=\&quot;Debian GNU/Linux 10 (buster)\&quot;&quot;,&quot;version&quot;:&quot;Kernel 5.15.0-48-generic&quot;}}}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2022-10-29T21:25:06.012+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;CONTROL&quot;,  &quot;id&quot;:21951,   &quot;ctx&quot;:&quot;initandlisten&quot;,&quot;msg&quot;:&quot;Options set by command line&quot;,&quot;attr&quot;:{&quot;options&quot;:{&quot;config&quot;:&quot;/opt/bitnami/mongodb/conf/mongodb.conf&quot;,&quot;net&quot;:{&quot;bindIp&quot;:&quot;*&quot;,&quot;ipv6&quot;:false,&quot;port&quot;:27017,&quot;unixDomainSocket&quot;:{&quot;enabled&quot;:true,&quot;pathPrefix&quot;:&quot;/opt/bitnami/mongodb/tmp&quot;}},&quot;processManagement&quot;:{&quot;fork&quot;:false,&quot;pidFilePath&quot;:&quot;/opt/bitnami/mongodb/tmp/mongodb.pid&quot;},&quot;replication&quot;:{&quot;enableMajorityReadConcern&quot;:true,&quot;replSetName&quot;:&quot;rs0&quot;},&quot;security&quot;:{&quot;authorization&quot;:&quot;disabled&quot;,&quot;keyFile&quot;:&quot;/opt/bitnami/mongodb/conf/keyfile&quot;},&quot;setParameter&quot;:{&quot;enableLocalhostAuthBypass&quot;:&quot;true&quot;},&quot;storage&quot;:{&quot;dbPath&quot;:&quot;/bitnami/mongodb/data/db&quot;,&quot;directoryPerDB&quot;:false,&quot;journal&quot;:{&quot;enabled&quot;:true}},&quot;systemLog&quot;:{&quot;destination&quot;:&quot;file&quot;,&quot;logAppend&quot;:true,&quot;logRotate&quot;:&quot;reopen&quot;,&quot;path&quot;:&quot;/opt/bitnami/mongodb/logs/mongodb.log&quot;,&quot;quiet&quot;:false,&quot;verbosity&quot;:0}}}}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2022-10-29T21:25:06.013+00:00&quot;},&quot;s&quot;:&quot;E&quot;,  &quot;c&quot;:&quot;STORAGE&quot;,  &quot;id&quot;:20557,   &quot;ctx&quot;:&quot;initandlisten&quot;,&quot;msg&quot;:&quot;DBException in initAndListen, terminating&quot;,&quot;attr&quot;:{&quot;error&quot;:&quot;DBPathInUse: Unable to create/open the lock file: /bitnami/mongodb/data/db/mongod.lock (Read-only file system). Ensure the user executing mongod is the owner of the lock file and has the appropriate permissions. Also make sure that another mongod instance is not already running on the /bitnami/mongodb/data/db directory&quot;}}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2022-10-29T21:25:06.013+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;REPL&quot;,     &quot;id&quot;:4784900, &quot;ctx&quot;:&quot;initandlisten&quot;,&quot;msg&quot;:&quot;Stepping down the ReplicationCoordinator for shutdown&quot;,&quot;attr&quot;:{&quot;waitTimeMillis&quot;:10000}}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2022-10-29T21:25:06.014+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;COMMAND&quot;,  &quot;id&quot;:4784901, &quot;ctx&quot;:&quot;initandlisten&quot;,&quot;msg&quot;:&quot;Shutting down the MirrorMaestro&quot;}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2022-10-29T21:25:06.014+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;SHARDING&quot;, &quot;id&quot;:4784902, &quot;ctx&quot;:&quot;initandlisten&quot;,&quot;msg&quot;:&quot;Shutting down the WaitForMajorityService&quot;}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2022-10-29T21:25:06.014+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;NETWORK&quot;,  &quot;id&quot;:20562,   &quot;ctx&quot;:&quot;initandlisten&quot;,&quot;msg&quot;:&quot;Shutdown: going to close listening sockets&quot;}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2022-10-29T21:25:06.014+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;NETWORK&quot;,  &quot;id&quot;:4784905, &quot;ctx&quot;:&quot;initandlisten&quot;,&quot;msg&quot;:&quot;Shutting down the global connection pool&quot;}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2022-10-29T21:25:06.014+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;STORAGE&quot;,  &quot;id&quot;:4784906, &quot;ctx&quot;:&quot;initandlisten&quot;,&quot;msg&quot;:&quot;Shutting down the FlowControlTicketholder&quot;}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2022-10-29T21:25:06.014+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;-&quot;,        &quot;id&quot;:20520,   &quot;ctx&quot;:&quot;initandlisten&quot;,&quot;msg&quot;:&quot;Stopping further Flow Control ticket acquisitions.&quot;}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2022-10-29T21:25:06.014+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;REPL&quot;,     &quot;id&quot;:4784907, &quot;ctx&quot;:&quot;initandlisten&quot;,&quot;msg&quot;:&quot;Shutting down the replica set node executor&quot;}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2022-10-29T21:25:06.014+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;NETWORK&quot;,  &quot;id&quot;:4784918, &quot;ctx&quot;:&quot;initandlisten&quot;,&quot;msg&quot;:&quot;Shutting down the ReplicaSetMonitor&quot;}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2022-10-29T21:25:06.014+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;SHARDING&quot;, &quot;id&quot;:4784921, &quot;ctx&quot;:&quot;initandlisten&quot;,&quot;msg&quot;:&quot;Shutting down the MigrationUtilExecutor&quot;}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2022-10-29T21:25:06.014+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;CONTROL&quot;,  &quot;id&quot;:4784925, &quot;ctx&quot;:&quot;initandlisten&quot;,&quot;msg&quot;:&quot;Shutting down free monitoring&quot;}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2022-10-29T21:25:06.014+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;STORAGE&quot;,  &quot;id&quot;:4784927, &quot;ctx&quot;:&quot;initandlisten&quot;,&quot;msg&quot;:&quot;Shutting down the HealthLog&quot;}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2022-10-29T21:25:06.014+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;STORAGE&quot;,  &quot;id&quot;:4784929, &quot;ctx&quot;:&quot;initandlisten&quot;,&quot;msg&quot;:&quot;Acquiring the global lock for shutdown&quot;}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2022-10-29T21:25:06.014+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;-&quot;,        &quot;id&quot;:4784931, &quot;ctx&quot;:&quot;initandlisten&quot;,&quot;msg&quot;:&quot;Dropping the scope cache for shutdown&quot;}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2022-10-29T21:25:06.014+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;FTDC&quot;,     &quot;id&quot;:4784926, &quot;ctx&quot;:&quot;initandlisten&quot;,&quot;msg&quot;:&quot;Shutting down full-time data capture&quot;}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2022-10-29T21:25:06.015+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;CONTROL&quot;,  &quot;id&quot;:20565,   &quot;ctx&quot;:&quot;initandlisten&quot;,&quot;msg&quot;:&quot;Now exiting&quot;}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2022-10-29T21:25:06.015+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;CONTROL&quot;,  &quot;id&quot;:23138,   &quot;ctx&quot;:&quot;initandlisten&quot;,&quot;msg&quot;:&quot;Shutting down&quot;,&quot;attr&quot;:{&quot;exitCode&quot;:100}}
</code></pre>
<p><strong>Update</strong></p>
<p>I checked the syslog and before the the logs <code>Nov 14 23:07:17 k8s-worker2 kubelet[752]: E1114 23:07:17.749057     752 pod_workers.go:951] &quot;Error syncing pod, skipping&quot; err=&quot;failed to \&quot;StartContainer\&quot; for \&quot;mongodb\&quot; with CrashLoopBackOff: \&quot;back-off 10s restarting failed container=mongodb pod=mongodb-2_mongodb(314f2776-ced4-4ba3-b90b-f927dc079770)\&quot;&quot; pod=&quot;mongodb/mongodb-2&quot; podUID=314f2776-ced4-4ba3-b90b-f927dc079770</code></p>
<p>I find these logs:</p>
<pre><code>Nov 14 23:06:59 k8s-worker2 kernel: [3413829.341806] sd 2:0:0:1: [sda] tag#42 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=11s
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.341866] sd 2:0:0:1: [sda] tag#42 Sense Key : Medium Error [current] 
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.341891] sd 2:0:0:1: [sda] tag#42 Add. Sense: Unrecovered read error
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.341899] sd 2:0:0:1: [sda] tag#42 CDB: Write(10) 2a 00 00 85 1f b8 00 00 40 00
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.341912] blk_update_request: critical medium error, dev sda, sector 8724408 op 0x1:(WRITE) flags 0x800 phys_seg 8 prio class 0
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.352012] Aborting journal on device sda-8.
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.354980] EXT4-fs error (device sda) in ext4_reserve_inode_write:5726: Journal has aborted
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.355103] sd 2:0:0:1: [sda] tag#40 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=15s
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.357056] sd 2:0:0:1: [sda] tag#40 Sense Key : Medium Error [current] 
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.357061] sd 2:0:0:1: [sda] tag#40 Add. Sense: Unrecovered read error
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.357066] sd 2:0:0:1: [sda] tag#40 CDB: Write(10) 2a 00 00 44 14 88 00 00 10 00
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.357068] blk_update_request: critical medium error, dev sda, sector 4461704 op 0x1:(WRITE) flags 0x800 phys_seg 2 prio class 0
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.357088] EXT4-fs error (device sda): ext4_dirty_inode:5922: inode #131080: comm mongod: mark_inode_dirty error
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.359566] EXT4-fs warning (device sda): ext4_end_bio:344: I/O error 7 writing to inode 131081 starting block 557715)
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.361432] EXT4-fs error (device sda) in ext4_dirty_inode:5923: Journal has aborted
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.362792] Buffer I/O error on device sda, logical block 557713
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.364010] Buffer I/O error on device sda, logical block 557714
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.365222] sd 2:0:0:1: [sda] tag#43 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=8s
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.365228] sd 2:0:0:1: [sda] tag#43 Sense Key : Medium Error [current] 
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.365230] sd 2:0:0:1: [sda] tag#43 Add. Sense: Unrecovered read error
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.365233] sd 2:0:0:1: [sda] tag#43 CDB: Write(10) 2a 00 00 44 28 38 00 00 08 00
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.365234] blk_update_request: critical medium error, dev sda, sector 4466744 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.367434] EXT4-fs warning (device sda): ext4_end_bio:344: I/O error 7 writing to inode 131083 starting block 558344)
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.367442] Buffer I/O error on device sda, logical block 558343
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.368593] sd 2:0:0:1: [sda] tag#41 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=15s
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.368597] sd 2:0:0:1: [sda] tag#41 Sense Key : Medium Error [current] 
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.368599] sd 2:0:0:1: [sda] tag#41 Add. Sense: Unrecovered read error
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.368602] sd 2:0:0:1: [sda] tag#41 CDB: Write(10) 2a 00 00 44 90 70 00 00 10 00
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.368604] blk_update_request: critical medium error, dev sda, sector 4493424 op 0x1:(WRITE) flags 0x800 phys_seg 2 prio class 0
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.370907] EXT4-fs warning (device sda): ext4_end_bio:344: I/O error 7 writing to inode 131081 starting block 561680)
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.370946] sd 2:0:0:1: [sda] tag#39 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=15s
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.370949] sd 2:0:0:1: [sda] tag#39 Sense Key : Medium Error [current] 
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.370952] sd 2:0:0:1: [sda] tag#39 Add. Sense: Unrecovered read error
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.370949] EXT4-fs error (device sda): ext4_journal_check_start:83: comm kworker/u4:0: Detected aborted journal
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.370954] sd 2:0:0:1: [sda] tag#39 CDB: Write(10) 2a 00 00 10 41 98 00 00 08 00
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.372081] blk_update_request: critical medium error, dev sda, sector 1065368 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.374353] EXT4-fs warning (device sda): ext4_end_bio:344: I/O error 7 writing to inode 131080 starting block 133172)
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.374396] Buffer I/O error on device sda, logical block 133171
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.388492] EXT4-fs error (device sda) in __ext4_new_inode:1136: Journal has aborted
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.390763] EXT4-fs error (device sda) in ext4_create:2786: Journal has aborted
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.391732] sd 2:0:0:1: [sda] tag#46 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.392941] sd 2:0:0:1: [sda] tag#46 Sense Key : Medium Error [current] 
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.392944] sd 2:0:0:1: [sda] tag#46 Add. Sense: Unrecovered read error
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.392948] sd 2:0:0:1: [sda] tag#46 CDB: Write(10) 2a 08 00 00 00 00 00 00 08 00
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.392950] blk_update_request: critical medium error, dev sda, sector 0 op 0x1:(WRITE) flags 0x23800 phys_seg 1 prio class 0
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.395562] Buffer I/O error on dev sda, logical block 0, lost sync page write
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.396945] sd 2:0:0:1: [sda] tag#45 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.396953] sd 2:0:0:1: [sda] tag#45 Sense Key : Medium Error [current] 
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.396955] sd 2:0:0:1: [sda] tag#45 Add. Sense: Unrecovered read error
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.396958] sd 2:0:0:1: [sda] tag#45 CDB: Write(10) 2a 08 00 84 00 00 00 00 08 00
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.396959] blk_update_request: critical medium error, dev sda, sector 8650752 op 0x1:(WRITE) flags 0x20800 phys_seg 1 prio class 0
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.396930] EXT4-fs (sda): I/O error while writing superblock
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.399771] Buffer I/O error on dev sda, logical block 1081344, lost sync page write
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.403897] JBD2: Error -5 detected when updating journal superblock for sda-8.
Nov 14 23:07:01 k8s-worker2 systemd[1]: run-docker-runtime\x2drunc-moby-d1c0f0dc3e024723707edfc12e023b98fb98f1be971177ecca5ac0cfdc91ab87-runc.w3zzIL.mount: Deactivated successfully.
Nov 14 23:07:05 k8s-worker2 kubelet[752]: E1114 23:07:05.415798     752 dns.go:157] &quot;Nameserver limits exceeded&quot; err=&quot;Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 46.38.252.230 46.38.225.230 2a03:4000:0:1::e1e6&quot;
Nov 14 23:07:06 k8s-worker2 kubelet[752]: E1114 23:07:06.412219     752 dns.go:157] &quot;Nameserver limits exceeded&quot; err=&quot;Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 46.38.252.230 46.38.225.230 2a03:4000:0:1::e1e6&quot;
Nov 14 23:07:06 k8s-worker2 systemd[1]: run-docker-runtime\x2drunc-moby-d1c0f0dc3e024723707edfc12e023b98fb98f1be971177ecca5ac0cfdc91ab87-runc.nK23K3.mount: Deactivated successfully.
Nov 14 23:07:11 k8s-worker2 systemd[1]: run-docker-runtime\x2drunc-moby-d1c0f0dc3e024723707edfc12e023b98fb98f1be971177ecca5ac0cfdc91ab87-runc.L5TkRU.mount: Deactivated successfully.
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.411831] sd 2:0:0:1: [sda] tag#44 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=15s
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.411888] sd 2:0:0:1: [sda] tag#44 Sense Key : Medium Error [current] 
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.411898] sd 2:0:0:1: [sda] tag#44 Add. Sense: Unrecovered read error
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.411952] sd 2:0:0:1: [sda] tag#44 CDB: Write(10) 2a 00 00 44 28 40 00 00 50 00
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.411965] blk_update_request: critical medium error, dev sda, sector 4466752 op 0x1:(WRITE) flags 0x0 phys_seg 10 prio class 0
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.419273] EXT4-fs warning (device sda): ext4_end_bio:344: I/O error 7 writing to inode 131083 starting block 558354)
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.430398] sd 2:0:0:1: [sda] tag#47 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=15s
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.430407] sd 2:0:0:1: [sda] tag#47 Sense Key : Medium Error [current] 
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.430409] sd 2:0:0:1: [sda] tag#47 Add. Sense: Unrecovered read error
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.430412] sd 2:0:0:1: [sda] tag#47 CDB: Write(10) 2a 08 00 00 00 00 00 00 08 00
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.430415] blk_update_request: critical medium error, dev sda, sector 0 op 0x1:(WRITE) flags 0x23800 phys_seg 1 prio class 0
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.433686] Buffer I/O error on dev sda, logical block 0, lost sync page write
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.436088] EXT4-fs (sda): I/O error while writing superblock
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.444291] sd 2:0:0:1: [sda] tag#32 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=14s
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.444300] sd 2:0:0:1: [sda] tag#32 Sense Key : Medium Error [current] 
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.444304] sd 2:0:0:1: [sda] tag#32 Add. Sense: Unrecovered read error
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.444308] sd 2:0:0:1: [sda] tag#32 CDB: Write(10) 2a 00 00 41 01 18 00 00 08 00
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.444313] blk_update_request: critical medium error, dev sda, sector 4260120 op 0x1:(WRITE) flags 0x3000 phys_seg 1 prio class 0
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.449491] Buffer I/O error on dev sda, logical block 532515, lost async page write
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.453591] sd 2:0:0:1: [sda] tag#33 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.453600] sd 2:0:0:1: [sda] tag#33 Sense Key : Medium Error [current] 
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.453603] sd 2:0:0:1: [sda] tag#33 Add. Sense: Unrecovered read error
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.453607] sd 2:0:0:1: [sda] tag#33 CDB: Write(10) 2a 08 00 00 00 00 00 00 08 00
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.453610] blk_update_request: critical medium error, dev sda, sector 0 op 0x1:(WRITE) flags 0x23800 phys_seg 1 prio class 0
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.459072] Buffer I/O error on dev sda, logical block 0, lost sync page write
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.461189] EXT4-fs (sda): I/O error while writing superblock
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.464347] EXT4-fs (sda): Remounting filesystem read-only
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.466527] EXT4-fs (sda): failed to convert unwritten extents to written extents -- potential data loss!  (inode 131081, error -30)
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.470833] Buffer I/O error on device sda, logical block 561678
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.473548] Buffer I/O error on device sda, logical block 561679
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.477384] EXT4-fs (sda): failed to convert unwritten extents to written extents -- potential data loss!  (inode 131083, error -30)
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.482014] Buffer I/O error on device sda, logical block 558344
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.484881] Buffer I/O error on device sda, logical block 558345
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.487224] Buffer I/O error on device sda, logical block 558346
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.488837] Buffer I/O error on device sda, logical block 558347
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.490543] Buffer I/O error on device sda, logical block 558348
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.492061] Buffer I/O error on device sda, logical block 558349
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.493494] Buffer I/O error on device sda, logical block 558350
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.494931] Buffer I/O error on device sda, logical block 558351
</code></pre>
<p>Not sure, if this is really related to the problem.</p>

<p>Generally when you see this error message:</p>
<pre><code>&quot;error&quot;:&quot;DBPathInUse: Unable to create/open the lock file: /bitnami/mongodb/data/db/mongod.lock (Read-only file system)
</code></pre>
<p>It most probably mean that your mongodb pod did not shutted down gracefully and had no time to remove the mongod.lock file so when your pod was re-created in another k8s node the &quot;new&quot; mongod process cannot start because it is finding the previous mongod.lock file.</p>
<p>The easiest way to resolve the current availability issue is to scale up and  add immediately one more replicaSet member so the new member to init-sync  from the available good member:</p>
<pre><code>helm upgrade mongodb bitnami/mongodb \
--set architecture=replicaset \
--set auth.replicaSetKey=myreplicasetkey \
--set auth.rootPassword=myrootpassword \
--set replicaCount=3
</code></pre>
<p>and elect again primary.</p>
<p>You can check if mongoDB replicaSet elected PRIMARY from mongo shell inside the pod with the command:</p>
<pre><code> rs.status()
</code></pre>
<p>For affected pod with the issue you can do as follow:</p>
<p>You can plan maitenance window and scale down ( scaling down stateFullset do not expect to automatically delete the <a href="https://kubernetes.io/docs/tasks/administer-cluster/change-pv-reclaim-policy/" rel="nofollow noreferrer">pvc/pv</a> , but good to make backup just in case.</p>
<p>After you scale down you can start custom helper pod to mount the pv so you can remove the mongod.lock file:</p>
<p>Temporary pod that you will start to mount the affected dbPath  and remove the mongodb.lock file:</p>
<pre><code>   kubectl apply -f - &lt;&lt;EOF
   apiVersion: v1
   kind: Pod
   metadata:
     name: mongo-pvc-helper
   spec:
     securityContext:
       runAsUser: 0
    containers:
    - command:
       - sh
       - -c
       - while true ; do echo alive ; sleep 10 ; done
    image: busybox
    imagePullPolicy: Always
    name: mongo-pvc-helper
    resources: {}
      securityContext:
        capabilities:
          drop:
            - ALL
   volumeMounts:
    - mountPath: /mongodata
        name: mongodata
   volumes:
     - name: mongodata
       persistentVolumeClaim:
         claimName: &lt;your_faulty_pod_pvc_name&gt;
  EOF
</code></pre>
<p>After you start the pod you can do:</p>
<pre><code>kubectl exec mongo-pvc-helper -it sh
$ chown -R 0:0 /mongodata
$ rm /mongodata/mongod.lock
$ exit
</code></pre>
<p>Or you can complete wipe up the entire pv(if you prefer safely to init-sync entirely this member):</p>
<pre><code>rm -rf /mongodata/*
</code></pre>
<p>And terminate the pod so you can finish the process:</p>
<pre><code> kubectl delete pod mongo-pvc-helper
</code></pre>
<p>And again scale-up:</p>
<pre><code>  helm upgrade mongodb bitnami/mongodb \
  --set architecture=replicaset \
  --set auth.replicaSetKey=myreplicasetkey \
  --set auth.rootPassword=myrootpassword \
  --set replicaCount=2
</code></pre>
<p>Btw, good to have at least 3x  data members in replicaSet  for better redundancy to allow during single member down event election to keep still the PRIMARY up and running...</p>
<p>How to troubleshoot this further:</p>
<ol>
<li><p>Ensure your pods have the terminationGracePeriod set (at least 10-20 sec) so it allow some time for the mongod process to flush data to storage and remove the mongod.lock file.</p>
</li>
<li><p>Depending from pod memory limits/requests , you can set some safer value for storage.wiredTiger.engineConfig.cacheSizeGB (if not set it is allocating ~50% from memory  ).</p>
</li>
<li><p>Check the kubelet logs from node where pod was killed there maybe more details why pod was killed.</p>
</li>
</ol>
