<p>I am trying to use Velero as backup and disaster recovery tool in Google Cloud Platform with multiple GCP regions (for example: europe-north1 and europe-west4) for GKE private clusters. I was able to successfully backup and restore using velero in the same region (taking backup of gke cluster in europe-north1 and restoring to another gke cluster in europe-north1) without any issues. This works fine because the snapshots are stored in the same region (europe-north1) for both the clusters.</p>
<p>But I would like to use velero as the disaster recovery tool for the GKE clusters so that I can take backup of GKE clusters in europe-north1 region and restore them to europe-west4 region. On further research, I found that by <a href="https://velero.io/blog/csi-integration/" rel="nofollow noreferrer">enabling CSI plugin</a> support for velero, I would be able to achieve the same. So I have folowed the guidelines to use CSI plugin with velero but I'm still not able to restore the persistent disk PVCs to another region. The snapshots are taken as multi-regional (for example, eu). But when I run the velero restore command, pod creation (I am using wordpress and mysql pods as examples) has been in 'pending' state.</p>
<p>kubectl describe pod (mysql and wordpress) gives the following error:</p>
<pre><code>   Normal   NotTriggerScaleUp  72s (x31 over 6m12s)  cluster-autoscaler  pod didn't trigger scale-up (it wouldn't fit if a new node is added):
  Warning  FailedScheduling   60s (x11 over 6m14s)  default-scheduler   0/4 nodes are available: 4 node(s) had volume node affinity conflict.
</code></pre>
<p>This error is because the google persistent disks which are created by PVC are in a different region than the GKE Cluster. Checking the disks, I can see that the restore command created two disks but they are still created in europe-north1 region (primary gke cluster region) instead of being created in europe-west4 region where the secondary gke cluster resides.</p>
<p>As this is a new feature for velero (CSI Plugin), I couldn't find any documentation for using it in GCP (there is a document showing <a href="https://velero.io/blog/csi-integration/" rel="nofollow noreferrer">CSI implementation with Azure Disks</a>)</p>
<p>Minimum requirement for CSI Plugin to work with velero backups:</p>
<pre><code>kubernetes version : 1.17
velero version: 1.4.2
</code></pre>
<p>Velero Version</p>
<pre><code>velero version
Client:
        Version: v1.4.2
        Git commit: 56a08a4d695d893f0863f697c2f926e27d70c0c5
Server:
        Version: v1.4.2
</code></pre>
<p>GKE Cluster kubernetes version (GKE cluster created with GcePersistentDiskCsiDriver=ENABLED addon) :</p>
<pre><code>v1.17.9-gke.600 
</code></pre>
<p>Primary Region:</p>
<pre><code>europe-north1
</code></pre>
<p>Secondary (DR) Region:</p>
<pre><code>europe-west4
</code></pre>
<p>Command used to install velero server (with CSI plugin enabled):</p>
<pre><code>velero install \ 
--features=EnableCSI \ 
--provider=gcp \ 
--image=gcr.io/$project/velero:v1.4.2  \ 
--plugins=gcr.io/$project/velero-plugin-for-gcp:v1.1.0,gcr.io/$project/velero-plugin-for-csi:v0.1.0 \ 
--bucket=$storagebucket \ 
--secret-file=$HOME/./velero-backup-storage-sa-key.json
</code></pre>
<p>Other documents that I have referred for this:</p>
<p><a href="https://velero.io/docs/v1.4/csi/#installing-velero-with-csi-support" rel="nofollow noreferrer">https://velero.io/docs/v1.4/csi/#installing-velero-with-csi-support</a></p>
<p><a href="https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/gce-pd-csi-driver" rel="nofollow noreferrer">https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/gce-pd-csi-driver</a></p>
<p>Any help would be much appreciated.</p>

<p>In GCP, VolumeSnapshots are multi-region by default (but single geo - us, europe or asia). I have successfully tested a cross region restore for a StatefulSet from us-central1 to us-east4 in GCP. But one caveat is that I used Regional Disks and created region/zone specific StorageClasses with <code>allowedTopologies</code> configured. Here is my restore SC:</p>
<pre class="lang-yaml prettyprint-override"><code>kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: &quot;true&quot;
  name: regional-pd-ssd-csi-storageclass
provisioner: pd.csi.storage.gke.io
parameters:
  type: pd-ssd
  replication-type: regional-pd
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowedTopologies:
- matchLabelExpressions:
  - key: topology.gke.io/zone
    values:
    - us-east4-b
    - us-east4-c
</code></pre>
<p>And here is the SC for the backup region:</p>
<pre class="lang-yaml prettyprint-override"><code>kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: &quot;true&quot;
  name: regional-pd-ssd-csi-storageclass
provisioner: pd.csi.storage.gke.io
parameters:
  type: pd-ssd
  #replication-type is one of none or regional-pd, defaults to none (zonal PD)
  replication-type: regional-pd
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowedTopologies:
- matchLabelExpressions:
  - key: topology.gke.io/zone
    values:
    - us-central1-a
    - us-central1-b
</code></pre>
<p>It may be possible to use <code>allowedTopologies</code> without regional disks but that parameters is not supported by the EBS  CSI driver (as far as I know).</p>
