<p>Recently, the managed pod in my mongo deployment onto GKE was automatically deleted and a new one was created in its place. As a result, all my db data was lost.</p>
<p>I specified a PV for the deployment and the PVC was bound too, and I used the standard storage class (google persistent disk). The Persistent Volume Claim had not been deleted either.</p>
<p>Here's an image of the result from <code>kubectl get pv</code>:
<a href="https://i.stack.imgur.com/emsMe.png" rel="nofollow noreferrer">pvc</a></p>
<p>My mongo deployment along with the persistent volume claim and service deployment were all created by using kubernets' <code>kompose</code> tool from a docker-compose.yml for a <a href="https://v1.prisma.io/docs/1.34/prisma-server/local-prisma-setup-je3i/" rel="nofollow noreferrer">prisma 1 + mongodb</a> deployment.</p>
<p>Here are my yamls:</p>
<p><code>mongo-deployment.yaml</code></p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    kompose.cmd: kompose -f docker-compose.yml convert
    kompose.version: 1.21.0 (992df58d8)
  creationTimestamp: null
  labels:
    io.kompose.service: mongo
  name: mongo
  namespace: dbmode
spec:
  replicas: 1
  selector:
    matchLabels:
      io.kompose.service: mongo
  strategy:
    type: Recreate
  template:
    metadata:
      annotations:
        kompose.cmd: kompose -f docker-compose.yml convert
        kompose.version: 1.21.0 (992df58d8)
      creationTimestamp: null
      labels:
        io.kompose.service: mongo
    spec:
      containers:
      - env:
        - name: MONGO_INITDB_ROOT_PASSWORD
          value: prisma
        - name: MONGO_INITDB_ROOT_USERNAME
          value: prisma
        image: mongo:3.6
        imagePullPolicy: &quot;&quot;
        name: mongo
        ports:
        - containerPort: 27017
        resources: {}
        volumeMounts:
        - mountPath: /var/lib/mongo
          name: mongo
      restartPolicy: Always
      serviceAccountName: &quot;&quot;
      volumes:
      - name: mongo
        persistentVolumeClaim:
          claimName: mongo
status: {}
</code></pre>
<p><code>mongo-persistentvolumeclaim.yaml</code></p>
<pre><code>apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  creationTimestamp: null
  labels:
    io.kompose.service: mongo
  name: mongo
  namespace: dbmode
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 100Mi
status: {}

</code></pre>
<p><code>mongo-service.yaml</code></p>
<pre><code>apiVersion: v1
kind: Service
metadata:
  annotations:
    kompose.cmd: kompose -f docker-compose.yml convert
    kompose.version: 1.21.0 (992df58d8)
  creationTimestamp: null
  labels:
    io.kompose.service: mongo
  name: mongo
  namespace: dbmode
spec:
  ports:
  - name: &quot;27017&quot;
    port: 27017
    targetPort: 27017
  selector:
    io.kompose.service: mongo
status:
  loadBalancer: {}
</code></pre>
<p>I've tried checking the contents mounted in <code>/var/lib/mongo</code> and all I got was an empty <code>lost+found/</code> folder, and I've tried to search the Google Persistent Disks but there was nothing in the root directory and I didn't know where else to look.</p>
<p>I guess that for some reason the mongo deployment is not pulling from the persistent volume for the old data when it starts a new pod, which is extremely perplexing.</p>
<p>I also have another kubernetes project where the same thing happened, except that the old pod still showed but had an <code>evicted</code> status.</p>

Jonathan Lynn
<blockquote>
<p>I've tried checking the contents mounted in /var/lib/mongo and all I
got was an empty lost+found/ folder,</p>
</blockquote>
<p>OK, but have you checked it was actually saving data there, before the <code>Pod</code> restart and data loss ? I guess it was never saving any data in that directory.</p>
<p>I checked the image you used by running a simple <code>Pod</code>:</p>
<pre><code>apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
  - name: my-pod
    image: mongo:3.6
</code></pre>
<p>When you connect to it by running:</p>
<pre><code>kubectl exec -ti my-pod -- /bin/bash
</code></pre>
<p>and check the default mongo configuration file:</p>
<pre><code>root@my-pod:/var/lib# cat /etc/mongod.conf.orig
# mongod.conf

# for documentation of all options, see:
#   http://docs.mongodb.org/manual/reference/configuration-options/

# Where and how to store data.
storage:
  dbPath: /var/lib/mongodb # 👈
  journal:
    enabled: true
#  engine:
#  mmapv1:
#  wiredTiger:
</code></pre>
<p>you can see among other things that <code>dbPath</code> is actually set to <code>/var/lib/mongodb</code> and <strong>NOT</strong> to <code>/var/lib/mongo</code>.</p>
<p>So chances are that your mongo wasn't actually saving any data to your <code>PV</code> i.e. to <code>/var/lib/mongo</code> directory, where it was mounted, but to <code>/var/lib/mongodb</code> as stated in its configuration file.</p>
<p>You should be able to check it easily by <code>kubectl exec</code> to your running mongo pod:</p>
<pre><code>kubectl exec -ti &lt;mongo-pod-name&gt; -- /bin/bash
</code></pre>
<p>and verify where the data is saved.</p>
<p>If you didn't overwrite in any way the original config file (e.g. by providing a <code>ConfigMap</code>), <code>mongo</code> should save its data to <code>/var/lib/mongodb</code> and this directory, not being a mount point for your volume, is part of a <code>Pod</code> filesystem and its ephemeral.</p>
<h3>Update:</h3>
<p>The above mentioned <code>/etc/mongod.conf.orig</code> is only a template so it doesn't reflect the actual configuration that has been applied.</p>
<p>If you run:</p>
<pre><code>kubectl logs your-mongo-pod
</code></pre>
<p>it will show where the data directory is located:</p>
<pre><code>$ kubectl logs my-pod 
2020-12-16T22:20:47.472+0000 I CONTROL  [initandlisten] MongoDB starting : pid=1 port=27017 dbpath=/data/db 64-bit host=my-pod
2020-12-16T22:20:47.473+0000 I CONTROL  [initandlisten] db version v3.6.21
...
</code></pre>
<p>As we can see, data is saved in <code>/data/db</code>:</p>
<pre><code>dbpath=/data/db
</code></pre>
