<p>I want to create large EBS volumes (e.g. 1TB) dynamically for each jenkins worker pod on my EKS cluster such that each <code>persistentVolume</code> representing an EBS volume only persists for the lifespan of the pod.</p>
<p>I can't change the size of the nodes on the EKS cluster,
so I'm trying to use external EBS volumes via the <a href="https://github.com/kubernetes-sigs/aws-ebs-csi-driver" rel="nofollow noreferrer">ebs-csi-driver</a> helm chart.</p>
<p>I can't seem to find a configuration for the <code>podTemplate</code> that enables dynamic provisioning of the <code>persistentVolumeClaim</code> and subsequent EBS <code>persistentVolume</code> for hosting my builds. The only way I'm able to successfully provision and mount EBS <code>persistentVolumes</code> dynamically to my worker pods is by using my own <code>persistentVolumeClaim</code> which I have to reference in the <code>podTemplate</code> and manually deploy using <code>kubectl apply -f pvc.yaml</code> and destroy <code>kubectl delete -f pvc.yaml</code> for each build cycle.</p>
<p>So far I've been able to provision one <code>persistentVolume</code> at a time using the following:</p>
<ul>
<li><code>persistentVolumeClaim</code>:
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ebs-claim
  namespace: jenkins
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: ebs-sc
  resources:
    requests:
      storage: 1000Gi 
</code></pre>
</li>
<li><code>StorageClass</code>:
<pre class="lang-yaml prettyprint-override"><code>apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-sc
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
</code></pre>
</li>
<li><code>podTemplate</code>:
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: Pod
spec:
  serviceAccount: jenkins-sa
  securityContext:
    fsGroup: 1000
  containers:
  - name: jenkins-worker
    image: &lt;some-image&gt;
    imagePullPolicy: Always
    command:
    - cat
    tty: true
    securityContext:
      runAsGroup: 1000
      runAsUser: 1000
    volumeMounts:
    - name: build-volume
      mountPath: &lt;some-mount-path&gt;
  volumes:
  - name: build-volume
    persistentVolumeClaim:
      claimName: ebs-claim
</code></pre>
<ul>
<li>I annotated the &quot;jenkins-sa&quot; <code>serviceAccount</code> with an AWS IAM Role ARN (<code>eks.amazonaws.com/role-arn</code>)
with all the AWS IAM policies and trust-relationships needed to allow the jenkins worker Pod
to provision an AWS EBS <code>persistentVolume</code> (as described in the <a href="https://docs.aws.amazon.com/eks/latest/userguide/csi-iam-role.html" rel="nofollow noreferrer">AWS ebs-csi-driver docs</a>).</li>
</ul>
</li>
</ul>
<p>Once the pipeline is started, and the jenkins worker pod is executing a build,
I'm able to ssh into it and confirm that the external EBS volume is mouted to the build directory using <code>df</code>.
Happy days!</p>
<p>However, I noticed the <code>persistentVolume</code> provisioned by the jenkins worker pod was lingering around
after the build finished and the pod destroyed. Apparently this is because the <code>PersistentVolumeClaim</code>
needs to be deleted before the <code>persistentVolume</code> it's bound to can be released (see <a href="https://saturncloud.io/blog/kubernetes-the-correct-way-to-delete-persistent-volumes-pv/" rel="nofollow noreferrer">here</a> for details).</p>
<p>After some digging it looks like I need to specify a <code>dynamicPVC()</code> in either the
<code>volumes</code> or <code>workspaceVolume</code> spec in the <code>podTemplate</code> yaml.
However I'm struggling to find documentation around dynamic <code>PersistentVolumeClaim</code> provisioning
beyond the <a href="https://plugins.jenkins.io/kubernetes/" rel="nofollow noreferrer">jenkins kubernetes plugin page</a>.
Unfortunately when I try to create <code>persistentVolumeClaims</code> dynamically like this
it doesn't work and just uses the maximum storage that the node can provision (which is limited to 20Gi).</p>
<ul>
<li>Updated <code>podTemplate</code>:
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: Pod
spec:
  serviceAccount: jenkins-sa
  workspaceVolume:
    dynamicPVC:
      accessModes: ReadWriteOnce
      requestsSize: 800Gi
      storageClassName: ebs-sc
  securityContext:
    fsGroup: 1000
  containers:
  - name: jenkins-worker
    image: &lt;some-image&gt;
    imagePullPolicy: Always
    command:
    - cat
    tty: true
    securityContext:
      runAsGroup: 1000
      runAsUser: 1000
</code></pre>
</li>
</ul>
<p>I am expecting a new <code>PersistentVolumeClaim</code> and <code>PersistentVolume</code> to be created
by the <code>podTemplate</code> when I start my build pipeline, and subsequently destroyed
when the pipeline finishes and the pod released.</p>

nyserq
<p>you use k8s 1.23+, then this is possible for <code>StatefulSets</code>, i.e if you can change pods by <code>StatefulSets</code>.</p>
<p>The <a href="https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#persistentvolumeclaim-retention" rel="nofollow noreferrer">PersistentVolumeClaim retention</a> feature configures the volume retention behavior that applies when the StatefulSet is deleted with the field <code>whenDeleted</code></p>
<p>So when PersistentVolumeClaims (PVCs) generated from the StatefulSet spec template will be deleted automatically when the <code>StatefulSet is deleted</code> or pods in the StatefulSet are scaled down.</p>
<pre><code>kind: StatefulSet
...
spec:
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Delete
    whenScaled: Delete
...
</code></pre>
<p>Also, add the <code>reclaimPolicy: Delete</code> field in the <code>StorageClass</code> resource which determines what happens to the Persistent Volumes (PVs) associated with that StorageClass when they are dynamically provisioned and later released.</p>
<p>As a result:</p>
<ul>
<li>When a PV associated with this StorageClass is dynamically provisioned and later released, the PV and the underlying storage resources are automatically deleted.</li>
<li>This means that when the associated PVC is deleted, both the PV and the underlying storage are automatically cleaned up.</li>
</ul>
