<p>On GKE I created a <code>statefulset</code> containing a <code>volumeClaimTemplates</code>. Then all the related <code>PersistentVolumesClaims</code>, <code>PersistentVolumes</code> and <code>Google Persistent Disks</code> are automatically created:</p>
<pre><code>kubectl get pvc
NAME                           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
        76m
qserv-data-qserv-worker-0      Bound    pvc-c5e060dc-88cb-4630-8229-c4b1fcb4f64b   3Gi        RWO            qserv          76m
qserv-data-qserv-worker-1      Bound    pvc-5dfffc24-165c-4e2c-a1fa-fa11dd45616f   3Gi        RWO            qserv          76m
qserv-data-qserv-worker-2      Bound    pvc-14aa9a63-fae0-4328-aaaa-17db2dee4b79   3Gi        RWO            qserv          76m
qserv-data-qserv-worker-3      Bound    pvc-8b701396-42ab-4d15-8b68-9b03ce5a2d07   3Gi        RWO            qserv          76m
qserv-data-qserv-worker-4      Bound    pvc-7c49e7a0-fd73-467d-b677-820d899f41ee   3Gi        RWO            qserv          76m
</code></pre>
<pre><code>kubectl get pv
pvc-14aa9a63-fae0-4328-aaaa-17db2dee4b79   3Gi        RWO            Retain           Bound      default/qserv-data-qserv-worker-2      qserv                   77m
pvc-5dfffc24-165c-4e2c-a1fa-fa11dd45616f   3Gi        RWO            Retain           Bound      default/qserv-data-qserv-worker-1      qserv                   77m
pvc-7c49e7a0-fd73-467d-b677-820d899f41ee   3Gi        RWO            Retain           Bound      default/qserv-data-qserv-worker-4      qserv                   77m
pvc-8b701396-42ab-4d15-8b68-9b03ce5a2d07   3Gi        RWO            Retain           Bound      default/qserv-data-qserv-worker-3      qserv                   77m
pvc-c5e060dc-88cb-4630-8229-c4b1fcb4f64b   3Gi        RWO            Retain           Bound      default/qserv-data-qserv-worker-0      qserv                   77m
</code></pre>
<pre><code>gcloud compute disks list                                                                                                                   
NAME                                           LOCATION       LOCATION_SCOPE  SIZE_GB  TYPE         STATUS
...
pvc-14aa9a63-fae0-4328-aaaa-17db2dee4b79       us-central1-c  zone            3        pd-balanced  READY
pvc-5dfffc24-165c-4e2c-a1fa-fa11dd45616f       us-central1-c  zone            3        pd-balanced  READY
pvc-7c49e7a0-fd73-467d-b677-820d899f41ee       us-central1-c  zone            3        pd-balanced  READY
pvc-8b701396-42ab-4d15-8b68-9b03ce5a2d07       us-central1-c  zone            3        pd-balanced  READY
pvc-c5e060dc-88cb-4630-8229-c4b1fcb4f64b       us-central1-c  zone            3        pd-balanced  READY
</code></pre>
<p>Is there a simple way to extract PVC/PV yaml file so that I can re-create all PVs/PVCs using the same Google Disks. (This might be useful to move the data to a new GKE cluster in case I delete the current one, or to restore the data if somebody remove accidentally the PVCs/PVs)</p>
<pre><code>kubectl get pv,pvc -o yaml &gt; export.yaml
</code></pre>
<p>Above command does not work because there is too much technical fields set at runtime which prevent <code>kubectl apply -f export.yaml</code> to work. Would you know a way to remove these fields from <code>export.yaml</code></p>

Fabrice Jammes
<p>As asked in the question:</p>
<blockquote>
<p>Is there a simple way to extract PVC/PV yaml file so that I can re-create all PVs/PVCs using the same Google Disks.</p>
</blockquote>
<p>Some scripting would be needed to extract a manifest that could be used without any hassles.</p>
<p>I found a StackOverflow thread about similar question (how to export manifests):</p>
<ul>
<li><em><a href="https://stackoverflow.com/questions/61392206/kubectl-export-is-deprecated-any-alternative">Stackoverflow.com: Questions: Kubectl export is deprecated any alternative</a></em></li>
</ul>
<blockquote>
<p>A side note!</p>
<p>I also stumbled upon <a href="https://github.com/itaysk/kubectl-neat" rel="nofollow noreferrer">kubectl neat</a> (a plugin for <code>kubectl</code>) which will be referenced later in that answer.</p>
<p>As correctly pointed by the author of the post, <code>kubectl neat</code> will show the message at the time of installation:</p>
<blockquote>
<p><strong>WARNING: You installed plugin &quot;neat&quot; from the krew-index plugin repository.</strong></p>
<p><strong>These plugins are not audited for security by the Krew maintainers.</strong>
<strong>Run them at your own risk.</strong></p>
</blockquote>
</blockquote>
<p>I would consider going with some backup solution as more a viable option due to the fact of data persistence and in general data protection in case of any failure.</p>
<p>From backup solution side you can look here:</p>
<ul>
<li><em><a href="https://portworx.com/how-to-migrate-stateful-applications-from-one-gcp-region-to-another-with-portworx-kubemotion/" rel="nofollow noreferrer">Portwortx.com: How to migrate stateful application from one gcp region to another with portwortx kubemotion</a></em></li>
<li><em><a href="https://velero.io/" rel="nofollow noreferrer">Velero.io</a></em></li>
</ul>
<p><code>PV</code>'s in <code>GKE</code> are in fact <code>Google Persistent Disks</code>. You can create a snapshot/image of a disk as a backup measure. You can also use this feature to test how your migration behaves:</p>
<ul>
<li><em><a href="https://cloud.google.com/compute/docs/disks/create-snapshots" rel="nofollow noreferrer">Cloud.google.com: Compute: Docs: Disks: Create snapshots</a></em></li>
<li><em><a href="https://cloud.google.com/compute/docs/images/create-delete-deprecate-private-images" rel="nofollow noreferrer">Cloud.google.com: Compute: Docs: Images: Create delete deprecate private images</a></em></li>
</ul>
<hr />
<p><strong>Please consider below example as a workaround.</strong></p>
<p>I've managed to migrate an example <code>Statefulset</code> from one cluster to another with data stored on <code>gce-pd</code>.</p>
<p>Once more, I encourage you to check this docs about using preexisting disks in <code>GKE</code> to create a <code>Statefulset</code>:</p>
<ul>
<li><em><a href="https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/preexisting-pd" rel="nofollow noreferrer">Cloud.google.com: Kubernetes Engine: Docs: How to: Persistent Volumes: Preexisting PD</a></em></li>
</ul>
<p>Assuming that you used manifest from official Kubernetes site:</p>
<ul>
<li><em><a href="https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/" rel="nofollow noreferrer">Kubernetes.io: Docs: Concepts: Workloads: Controllers: Statefulset</a></em></li>
</ul>
<p>You can migrate it to another cluster by:</p>
<ul>
<li><strong>Setting the <code>ReclaimPolicy</code> to <code>Retain</code> on each <code>PV</code> used</strong>. &lt;-- IMPORTANT</li>
<li>Using <code>kubectl neat</code> to extract needed manifests</li>
<li>Editing previously extracted manifests</li>
<li>Deleting the existing workload on old cluster</li>
<li>Creating a workload on new cluster</li>
</ul>
<h3>Setting the <code>ReclaimPolicy</code> to <code>Retain</code> on each <code>PV</code> used</h3>
<p>You will need to check if your <code>PV</code> <code>ReclaimPolicy</code> is set to <code>Retain</code>. This would stop <code>gce-pd</code> deletion after <code>PVC</code> and <code>PV</code> are deleted from the cluster. You can do it by following Kubernetes documentation:</p>
<ul>
<li><em><a href="https://kubernetes.io/docs/tasks/administer-cluster/change-pv-reclaim-policy/" rel="nofollow noreferrer">Kubernetes.io: Docs: Tasks: Administer cluster: Change PV reclaim policy</a></em></li>
</ul>
<p>More reference:</p>
<ul>
<li><em><a href="https://cloud.google.com/kubernetes-engine/docs/concepts/persistent-volumes#dynamic_provisioning" rel="nofollow noreferrer">Cloud.google.com: Kubernetes Engine: Docs: Concepts: Persistent Volumes: Dynamic provisioning</a></em></li>
</ul>
<h3>Using <code>kubectl neat</code> to extract needed manifests</h3>
<p>There are many ways to extract the manifest from Kubernetes API. I stumbled upon <code>kubectl neat</code> <a href="https://github.com/kubermatic/fubectl/pull/58#issuecomment-729546861" rel="nofollow noreferrer">here</a>. <a href="https://github.com/itaysk/kubectl-neat" rel="nofollow noreferrer">kubectl-neat</a> will remove some of the fields in the manifests.</p>
<p>I used it in a following manner:</p>
<ul>
<li><code>$ kubectl get statefulset -o yaml | kubectl neat &gt; final-sts.yaml</code></li>
<li><code>$ kubectl get pvc -o yaml | kubectl neat &gt; final-pvc.yaml</code></li>
<li><code>$ kubectl get pv -o yaml | kubectl neat &gt; final-pv.yaml</code></li>
</ul>
<blockquote>
<p>Disclaimer!</p>
<p>This workaround will use the names of the dynamically created disks in <code>GCP</code>. If you were to create new disks (from snapshot for example) you would need to modify whole setup (use preexsiting disks guide referenced earlier).</p>
</blockquote>
<p>Above commands would store manifests of a <code>StatefulSet</code> used in the Kubernetes examples.</p>
<h3>Editing previously extracted manifests</h3>
<p>You will need to edit this manifests to be used in newly created cluster. This part could be automated:</p>
<ul>
<li><code>final-pv.yaml</code> - delete the <code>.claimRef</code> in <code>.spec</code></li>
</ul>
<h3>Deleting the existing workload on old cluster</h3>
<p>You will need to release used disks so that the new cluster could use them. You will need to delete this <code>Statefulset</code> and accompanying <code>PVC</code>'s and <code>PV</code>'s. Please make sure that the <code>PV</code>'s <code>reclaimPolicy</code> is set to <code>Retain</code>.</p>
<h3>Creating a workload on new cluster</h3>
<p>You will need to use previously created manifest and apply them on a new cluster:</p>
<ul>
<li><code>$ kubectl apply -f final-pv.yaml</code></li>
<li><code>$ kubectl apply -f final-pvc.yaml</code></li>
<li><code>$ kubectl apply -f final-sts.yaml</code></li>
</ul>
<hr />
<p>As for exporting manifests you could also look (if feasible) on Kubernetes client libraries:</p>
<ul>
<li><em><a href="https://kubernetes.io/docs/reference/using-api/client-libraries/" rel="nofollow noreferrer">Kubernetes.io: Docs: Reference: Using API: Client libraries</a></em></li>
</ul>
