<p>I have created a Persistent Volume and Volume claim for an app I am working on in GKE. The claim and storage appear to be setup correctly, however, the data doesn't persist if the pod is restarted. I am able to save data initially and I can see the file in the pod, but it disappears after restarting it.</p>
<p>I had asked this question previously, but didn't include my .yaml files and received a sort of generic answer as a result so I decided to repost with the .yaml files hoping someone could look at them and tell me where I am going wrong. From everything I've seen, it looks like the problem is in the Persistent Volume as the claim looks exactly like everyone else's.</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: prod-api-meta-uploads-k8s
  namespace: default
  resourceVersion: &quot;4500192&quot;
  selfLink: /apis/apps/v1/namespaces/default/deployments/prod-api-meta-uploads-k8s
  uid: *******
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: prod-api-meta-uploads-k8s
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        gcb-build-id: *****
        gcb-trigger-id:****
      creationTimestamp: null
      labels:
        app: prod-api-meta-uploads-k8s
        app.kubernetes.io/managed-by: gcp-cloud-build-deploy
        app.kubernetes.io/name: prod-api-meta-uploads-k8s
        app.kubernetes.io/version: becdb864864f25d2dcde2e62a2f70501cfd09f19
    spec:
      containers:
      - image: bitbucket.org/api-meta-uploads-k8s@sha256:7766413c0d
        imagePullPolicy: IfNotPresent
        name: prod-api-meta-uploads-k8s-sha256-1
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /uploads/profileImages
          name: uploads-volume-prod
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - name: uploads-volume-prod
        persistentVolumeClaim:
          claimName: my-disk-claim-1
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: &quot;2020-09-08T21:00:40Z&quot;
    lastUpdateTime: &quot;2020-09-10T04:54:27Z&quot;
    message: ReplicaSet &quot;prod-api-meta-uploads-k8s-5c8f66f886&quot; has successfully progressed.
    reason: NewReplicaSetAvailable
    status: &quot;True&quot;
    type: Progressing
  - lastTransitionTime: &quot;2020-09-10T06:49:41Z&quot;
    lastUpdateTime: &quot;2020-09-10T06:49:41Z&quot;
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: &quot;True&quot;
    type: Available
  observedGeneration: 36
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1
</code></pre>
<p>** Volume Claim</p>
<pre><code>apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  creationTimestamp: &quot;2020-09-09T16:12:51Z&quot;
  finalizers:
  - kubernetes.io/pvc-protection
  name: uploads-volume-prod
  namespace: default
  resourceVersion: &quot;4157429&quot;
  selfLink: /api/v1/namespaces/default/persistentvolumeclaims/uploads-volume-prod
  uid: f93e6134
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 30Gi
  storageClassName: standard
  volumeMode: Filesystem
  volumeName: pvc-f93e6
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 30Gi
  phase: Bound
</code></pre>
<p>*** PVC</p>
<pre><code>apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  finalizers:
  - kubernetes.io/pvc-protection
  name: my-disk-claim-1
  namespace: default
  resourceVersion: &quot;4452471&quot;
  selfLink: /api/v1/namespaces/default/persistentvolumeclaims/my-disk-claim-1
  uid: d533702b
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi
  storageClassName: fast
  volumeMode: Filesystem
  volumeName: pvc-d533702b
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 50Gi
  phase: Bound
</code></pre>

<p>As you are using <a href="https://cloud.google.com/kubernetes-engine" rel="nofollow noreferrer">GKE</a> you don't need to prepare <code>PersistentVolume</code> and <code>PersistentVolumeClaim</code> manually (static provisioning) in relationship 1:1, as GKE can use <a href="https://kubernetes.io/docs/concepts/storage/dynamic-provisioning/" rel="nofollow noreferrer">Dynamic Volume Provisioning</a>.
It's good described in <a href="https://kubernetes.io/docs/concepts/storage/persistent-volumes/" rel="nofollow noreferrer">Persistent Volumes</a></p>
<blockquote>
<p>When none of the static PVs the administrator created match a user's PersistentVolumeClaim, the cluster may try to dynamically provision a volume specially for the PVC.</p>
</blockquote>
<p>In <code>GKE</code> on the beginning you have at least one <a href="https://kubernetes.io/docs/concepts/storage/storage-classes/" rel="nofollow noreferrer">storageclass</a> named <code>standard</code>. It also have <code>(default)</code> next to name.</p>
<pre><code>$ kubectl get sc
NAME                 PROVISIONER            AGE
standard (default)   kubernetes.io/gce-pd   110m
</code></pre>
<p>It means that if you won't specify <code>storageClassName</code> in your <code>PersistentVolumeClaim</code>, it will use <code>storageclass</code> which is set as <code>default</code>. In your YAMLs I can see that you have used <code>storageClassName: standard</code>. If you will check this <code>storageclass</code> you will se what <a href="https://kubernetes.io/docs/concepts/storage/persistent-volumes/#reclaiming" rel="nofollow noreferrer">ReclaimPolicy</a> was set to <code>delete</code>. Below output:</p>
<pre><code>$ kubectl describe sc standard
Name:                  standard
IsDefaultClass:        Yes
Annotations:           storageclass.kubernetes.io/is-default-class=true
Provisioner:           kubernetes.io/gce-pd
Parameters:            type=pd-standard
AllowVolumeExpansion:  True
MountOptions:          &lt;none&gt;
ReclaimPolicy:         Delete
VolumeBindingMode:     Immediate
Events:                &lt;none&gt;
</code></pre>
<p><code>IsDefaultClass</code>: means this <code>storageclass</code> is set as default.</p>
<p><code>ReclaimPolicy</code>: defining <code>ReclaimPolicy</code>, <code>delete</code> in this case</p>
<p>As ReclaimPolicy is set to <code>Delete</code>:</p>
<blockquote>
<p>For volume plugins that support the <code>Delete</code> reclaim policy, <strong>deletion removes both the <code>PersistentVolume</code> object from Kubernetes, as well as the associated storage asset in the external infrastructure, such as an AWS EBS, GCE PD, Azure Disk, or Cinder volume</strong>. Volumes that were dynamically provisioned inherit the reclaim policy of their <code>StorageClass</code>, which <strong>defaults to Delete</strong>. The administrator should configure the StorageClass according to users' expectations; otherwise, the PV must be edited or patched after it is created.</p>
</blockquote>
<p>Depends on your needs, you can use:</p>
<p><a href="https://kubernetes.io/docs/concepts/storage/persistent-volumes/#recycle" rel="nofollow noreferrer">Recycle</a>:</p>
<blockquote>
<p>If supported by the underlying volume plugin, the Recycle reclaim policy performs a basic scrub (rm -rf /thevolume/*) on the volume and makes it available again for a new claim.
However, please keep in mind that: <code>Warning: The Recycle reclaim policy is deprecated. Instead, the recommended approach is to use dynamic provisioning.</code> Adding this option as I didnt see what K8s version are you using, <strong>however it's not supported on GKE.</strong></p>
</blockquote>
<p><a href="https://kubernetes.io/docs/concepts/storage/persistent-volumes/#retain" rel="nofollow noreferrer">Retain</a>:</p>
<blockquote>
<p>The Retain reclaim policy allows for manual reclamation of the resource. When the PersistentVolumeClaim is deleted, the PersistentVolume still exists and the volume is considered &quot;released&quot;. But it is not yet available for another claim because the previous claimant's data remains on the volume.</p>
</blockquote>
<p>In addition, as you are using GKE, it supports only <code>Delete</code> and <code>Retain</code>.</p>
<pre><code>The StorageClass &quot;another-storageclass&quot; is invalid: reclaimPolicy: Unsupported value: &quot;Recycle&quot;: supported values: &quot;Delete&quot;, &quot;Retain&quot;
</code></pre>
<p>In addition, as you specfied <code>revisionHistoryLimit: 10</code>, pod after 10 restarts will be recreated, in that situation <code>pod</code>, <code>pv</code> and <code>pvc</code> will be deleted when <code>ReclaimPolicy</code> will be set as <code>delete</code>.</p>
<p><strong>Solution</strong></p>
<p>As easiest solution, you should create new <code>StorageClass</code> with <code>ReclaimPolicy</code> different from <code>Delete</code> and use it in your <code>PVC</code>.</p>
<pre><code>$ kubectl get sc,pv,pvc -A
NAME                                               PROVISIONER            AGE
storageclass.storage.k8s.io/another-storageclass   kubernetes.io/gce-pd   53s
storageclass.storage.k8s.io/standard (default)     kubernetes.io/gce-pd   130m

NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM             STORAGECLASS           REASON   AGE
persistentvolume/pvc-67c35c06-3f38-4f55-98c8-6b2b41ae5313   1Gi        RWO            Retain           Bound    tst-dev/pvc-1     another-storageclass            43s
persistentvolume/pvc-be30a43f-e96c-4c9f-8863-464823899a8f   1Gi        RWO            Retain           Bound    tst-stage/pvc-2   another-storageclass            42s

NAMESPACE   NAME                          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS           AGE
tst-dev     persistentvolumeclaim/pvc-1   Bound    pvc-67c35c06-3f38-4f55-98c8-6b2b41ae5313   1Gi        RWO            another-storageclass   46s
tst-stage   persistentvolumeclaim/pvc-2   Bound    pvc-be30a43f-e96c-4c9f-8863-464823899a8f   1Gi        RWO            another-storageclass   45s

$ kubectl delete pvc pvc-1 -n tst-dev
persistentvolumeclaim &quot;pvc-1&quot; deleted
user@cloudshell:~ (project)$ kubectl delete pvc pvc-2  -n tst-stage
persistentvolumeclaim &quot;pvc-2&quot; deleted
user@cloudshell:~ (project)$ kubectl get sc,pv,pvc -A
NAME                                               PROVISIONER            AGE
storageclass.storage.k8s.io/another-storageclass   kubernetes.io/gce-pd   7m49s
storageclass.storage.k8s.io/standard (default)     kubernetes.io/gce-pd   137m

NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM             STORAGECLASS           REASON   AGE
persistentvolume/pvc-67c35c06-3f38-4f55-98c8-6b2b41ae5313   1Gi        RWO            Retain           Released   tst-dev/pvc-1     another-storageclass            7m38s
persistentvolume/pvc-be30a43f-e96c-4c9f-8863-464823899a8f   1Gi        RWO            Retain           Released   tst-stage/pvc-2   another-storageclass            7m37s
</code></pre>
