<p>In a Kubernetes cluster, I'd like to be able to schedule a Job (with a CronJob) that will mount the same Volumes as 1 Pod of a given StatefulSet. Which Pod that is is a run time decision, depending on the labels set on the Pod at the time of scheduling the Job.</p>

<p>I guess many people will wonder why, therefore a description of what we're doing and trying to do:</p>

<h1>Current setup</h1>

<p>We have a StatefulSet which serves a PostgreSQL database. (one primary, multiple replica's)
We want to be able to create a backup from one of the pods of the StatefulSet.</p>

<p>For PostgreSQL we can already do backups over the network with <code>pg_basebackup</code>, however we are running multi-TB PostgreSQL databases, which means full streaming backups (with <code>pg_basebackup</code>) is not feasible.</p>

<p>We currently use <code>pgBackRest</code> to backup the databases, which allows for incremental backups.</p>

<p>As the incremental backup of <code>pgBackRest</code> requires access to the data volume and the WAL volume, we need to run the Backup Container on the same Kubernetes Node as the PostgreSQL instance, we currently even run it inside the same Pod in a separate Container.</p>

<p>Inside the container, a small api wraps around <code>pgBackRest</code> and can be triggered by sending <code>POST</code> requests to the api, this triggering is currently done using CronJobs.</p>

<h2>Downsides</h2>

<ul>
<li>Every PostgreSQL instance has multiple containers in the Pod, 1 to serve Postgres, 1 to serve a tiny wrapper around <code>pgBackRest</code></li>
<li>Job Logs only show successful backup triggers, the actual backup logs are part of the Backup container</li>
<li>The Pod that will run the backup may run on a relatively old configuration, changing backup configuration requires a rescheduling of the Pod, which may mean a fail over of the PostgreSQL primary.</li>
</ul>

<h1>Proposed setup</h1>

<p>Have a CronJob schedule a Pod that has the same Volume's as 1 of the Pods of the StatefulSet. This will allow the backup to use these Volumes.</p>

<p>However, which Volumes it needs is a run time decision: We may want to run the backup on the Volumes connected to the primary, or we may want to backup using the Volumes of a replica. The primary/replica may change at any moment, as auto-failover of the PostgreSQL primary is part of the solution.</p>

<p>Currently, this is not possible, as I cannot in the CronJob spec find any way to use information from the k8s api.</p>

<p>What does work, but is not very nice:</p>

<ul>
<li>Use a CronJob that schedules a Job</li>
<li>This Job queries the k8s api and schedules <em>another</em> job</li>
</ul>

<p>For example, this is what we can do to have a job create another job using this run-time information:</p>

<pre><code>apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: schedule-backup
spec:
  schedule: "13 03 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: backup-trigger
            image: bitnami/kubectl
            command:
            - sh
            - -c
            - |
              PRIMARYPOD=$(kubectl get pods -l cluster-name=&lt;NAME&gt;,role=master -o custom-columns=":metadata.name" --no-headers)
              kubectl apply -f - &lt;&lt;__JOB__
                apiVersion: batch/v1
                kind: Job
                metadata:
                  name: test
                spec:
                  volumes:
                    name: storage-volume
                    persistentVolumeClaim:
                      claimName:
                        data-volume-${PRIMARYPOD}
                  [...]
              __JOB__
</code></pre>

<p>The above may be best served by an Operator instead of using just a CronJob, but I'm wondering if anyone has a solution to the above.</p>

<h2>Downsides</h2>

<ul>
<li>Job Logs only show successful backup triggers, the actual backup logs are part of <em>another</em> Job</li>
<li>The Job requires permissions to schedule Pod's, requiring yet another role/rolebinding</li>
<li>Using heredocs in Bash makes things harder to read/parse/understand </li>
</ul>

<h1>Summary</h1>

<p>Long story, but these are the constraints we want to satisfy:</p>

<ul>
<li>Run a Backup of PostgreSQL database</li>
<li>These are multi-TB databases</li>
<li>Therefore, incremental backups are required</li>
<li>Therefore, we need to mount the same PV of an already running Pod</li>
<li>Therefore, we need to run a Pod (or container) on the same K8s Node as the PV</li>
<li>We want to be able to express this in a CronJob spec, instead of having to do runtime kubernetes api calls</li>
</ul>

Feike Steenbergen
<p>Well, simple and short answer would be: you generally can't.</p>

<p>But let's be creative for while :)</p>

<p>A very limited amount of storage backends support RWX (read write many) access, and in most cases these are the slower ones which you want to avoid when using for a database. This means that unless you run your backup wrapper as a sidecar (which you do now) you can't really access the PVs in a different POD, period.</p>

<p>I'd probably stick to your original approach, with some tweaks (like making sure that you never bring down primary due to backup / config change).</p>

<p>On a up to date K8S cluster and supported infrastructure provider, you could probably look into <a href="https://kubernetes.io/docs/concepts/storage/volume-snapshots/" rel="nofollow noreferrer">VolumeSnapshots</a> for snapshot based backups and potentially using the snapshot as source to spin up an incremental backup job. Sounds a bit convoluted though.</p>

<p>You could also run a backup dedicated postgres replica pod with limited resources (no live traffic) and embed backup logic only in that Pod.</p>
