<p>I want to use Velero to backup an application to a minio bucket. Here's some context, I have 2 AKS clusters [dev, tools].</p>
<p>The tools cluster runs my minio instance and dev is the cluster for my workloads.</p>
<p>I followed some examples on the internet on how to install Velero using helm and how to configure it to backup worloads to minio.</p>
<p>Right now, I can do a backup of an application with it's PersistentVolume but when I do a restore there's no data in the volume. I will go into detail below, and I appreciate any advice or help from the community to resolve this issue.</p>
<p>Here's the steps that I followed:</p>
<ol>
<li>Installing Velero :</li>
</ol>
<pre><code>helm repo add vmware-tanzu https://vmware-tanzu.github.io/helm-charts
helm repo update
helm upgrade --install $RELEASE_NAME vmware-tanzu/velero \ 
  --namespace $NAMESPACE --create-namespace -f $VALUES_FILE
</code></pre>
<ol start="2">
<li>Here's an extract of the helm values.yaml file I use, with the most important bits :</li>
</ol>
<pre><code>initContainers:
    - name: velero-plugin-for-aws
      image: velero/velero-plugin-for-aws:v1.6.1
      imagePullPolicy: IfNotPresent
      volumeMounts:
        - mountPath: /target
          name: plugins

  configuration:
    plugins:
      aws:
        image: velero/velero-plugin-for-aws:v1.2.0
      minio:
        image: velero/velero-plugin-for-minio:v1.2.0
  
    provider: aws
  
    backupStorageLocation:
      name: default 
      bucket: dev-velero-backup
      config:
        region: minio
        s3ForcePathStyle: &quot;true&quot;
        publicUrl: http://dev.api
        s3Url: &quot;minio.tenant&quot;
        insecureSkipTLSVerify: true
    
    volumeSnapshotLocation:
      region: minio
      name: default
      provider: aws
      config:
        region: minio
  
  # specify the credentials for minio.
  credentials:
    useSecret: true
    existingSecret: &quot;&quot;
    secretContents:
      cloud: |
        [default]
        aws_access_key_id = minio
        aws_secret_access_key = minio
      s3: &quot;&quot;
  
    features:
    namespace: velero
  backupsEnabled: true
  snapshotsEnabled: true
</code></pre>
<ol start="3">
<li><p>When I run the backup command I can see the objects created in the minio bucket, so there's not an issue with the communication between velero and minio.</p>
</li>
<li><p>This is the command I use to do a backup of my nginx-example application :
<code>velero backup create nginx-example --include-namespaces nginx-example --snapshot-volumes</code></p>
</li>
</ol>
<p>The backups complete without any errors.</p>
<p>Here are the logs from the backup :</p>
<pre><code>time=&quot;2023-03-21T13:11:28Z&quot; level=info msg=&quot;Executing RemapCRDVersionAction&quot; backup=velero/nginx-example cmd=/velero logSource=&quot;pkg/backup/remap_crd_version_action.go:61&quot; pluginName=velero
time=&quot;2023-03-21T13:11:28Z&quot; level=info msg=&quot;Exiting RemapCRDVersionAction, the cluster does not support v1beta1 CRD&quot; backup=velero/nginx-example cmd=/velero logSource=&quot;pkg/backup/remap_crd_version_action.go:89&quot; pluginName=velero
time=&quot;2023-03-21T13:11:28Z&quot; level=info msg=&quot;Backed up a total of 24 items&quot; backup=velero/nginx-example logSource=&quot;pkg/backup/backup.go:413&quot; progress=
</code></pre>
<ol start="5">
<li>The next step is simulating a DR event, by deleting the nginx-example namespace and verifying that all k8s resources for the app is destroyed including the PV.</li>
</ol>
<p><code>kubectl delete ns nginx-example</code>
<code>#Wait, and Check if pv is deleted.</code></p>
<ol start="6">
<li>When I attempt to restore the nginx-example from the velero backup, by running this command:</li>
</ol>
<p><code>velero restore create --from-backup nginx-example --include-namespaces nginx-example --restore-volumes</code></p>
<p>I can see in the restore logs the following messages :</p>
<pre><code>velero restore logs nginx-example-20230321141504
time=&quot;2023-03-21T13:15:06Z&quot; level=info msg=&quot;Waiting for all post-restore-exec hooks to complete&quot; logSource=&quot;pkg/restore/restore.go:596&quot; restore=velero/nginx-example-20230321141504
time=&quot;2023-03-21T13:15:06Z&quot; level=info msg=&quot;Done waiting for all post-restore exec hooks to complete&quot; logSource=&quot;pkg/restore/restore.go:604&quot; restore=velero/nginx-example-20230321141504
time=&quot;2023-03-21T13:15:06Z&quot; level=info msg=&quot;restore completed&quot; logSource=&quot;pkg/controller/restore_controller.go:545&quot; restore=velero/nginx-example-20230321141504
</code></pre>
<ol start="7">
<li>When I verify if the nginx acess-logs still contain the data of previous visits, it is empty :</li>
</ol>
<p><code>kubectl exec -it nginx-deploy-bf489bc5-8jrtz -- cat /var/log/nginx/access.log</code></p>
<p>The nginx-example application mounts a path  to  <code>/var/log/nginx</code> on the PV.</p>
<pre><code>    spec:
      volumes:
        - name: nginx-logs
          persistentVolumeClaim:
           claimName: nginx-logs
      containers:
      - image: nginx:stable
        name: nginx
        ports:
        - containerPort: 80
        volumeMounts:
          - mountPath: &quot;/var/log/nginx&quot;
            name: nginx-logs
            readOnly: false
</code></pre>
<p>The end goal should be to do a successful backup and restore of the nginx-example application with it's persistent volume that contains the access logs data.</p>
<p>I'll be really happy if this issue can be resolved with your help and of course I will provide any relevant information.</p>
<h3>Additional Information</h3>
<ol>
<li>VolumeSnapshotLocation</li>
</ol>
<pre><code>apiVersion: velero.io/v1
kind: VolumeSnapshotLocation
metadata:
  annotations:
    helm.sh/hook: post-install,post-upgrade,post-rollback
    helm.sh/hook-delete-policy: before-hook-creation
  creationTimestamp: &quot;2023-03-21T01:26:37Z&quot;
  generation: 1
  labels:
    app.kubernetes.io/instance: velero-ontwikkel
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: velero
    helm.sh/chart: velero-3.1.4
  name: default
  namespace: velero
  resourceVersion: &quot;83378185&quot;
  uid: cea663dd-c1d9-4035-8c84-79a240f4351c
spec:
  config:
    region: minio
  provider: aws
</code></pre>
<ol start="2">
<li>Installed velero plugins</li>
</ol>
<pre><code>NAME                                        KIND
velero.io/crd-remap-version                 BackupItemAction
velero.io/pod                               BackupItemAction
velero.io/pv                                BackupItemAction
velero.io/service-account                   BackupItemAction
velero.io/aws                               ObjectStore
velero.io/add-pv-from-pvc                   RestoreItemAction
velero.io/add-pvc-from-pod                  RestoreItemAction
velero.io/admission-webhook-configuration   RestoreItemAction
velero.io/apiservice                        RestoreItemAction
velero.io/change-pvc-node-selector          RestoreItemAction
velero.io/change-storage-class              RestoreItemAction
velero.io/cluster-role-bindings             RestoreItemAction
velero.io/crd-preserve-fields               RestoreItemAction
velero.io/init-restore-hook                 RestoreItemAction
velero.io/job                               RestoreItemAction
velero.io/pod                               RestoreItemAction
velero.io/pod-volume-restore                RestoreItemAction
velero.io/role-bindings                     RestoreItemAction
velero.io/service                           RestoreItemAction
velero.io/service-account                   RestoreItemAction
velero.io/aws                               VolumeSnapshotter

</code></pre>

Chesneycar
<p>you may need to enable NodeAgent to do volume backups，add deployNodeAgent: true in helm values.yaml, and when do backups use option --default-volumes-to-fs-backup(like old version using option --default-volumes-to-restic).</p>
<pre><code>velero backup create backup-test --include-namespaces nginx-example --default-volumes-to-fs-backup --snapshot-volumes --ttl 180h
</code></pre>
<p>after done backup you could do describe with --details and you could find something looks like this and you will know volume backup success.
velero backup describe backup-test --details</p>
<pre><code>kopia Backups:
  Completed:
    nginx-example/nginx-deployment-5b47dbff44-cw9l4: nginx-logs
</code></pre>
