<p>We have <a href="https://github.com/jonashackt/tekton-argocd-eks" rel="noreferrer">a EKS setup provisioned</a> where we use a Cloud Native Buildpacks based Tekton Pipeline our <code>PipelineRuns</code> are stuck and pending forever without getting a Pod resource. We created a PersistentVolumeClaim like this:</p>
<pre><code>apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: buildpacks-source-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 500Mi
</code></pre>
<p>Having a look into the events of this PVC after creation there are the following events indicating something is wrong with our EKS setup:</p>
<p>The command <code>kubectl describe pvc buildpacks-source-pvc</code> gives the following event messages:</p>
<pre><code>Name:          buildpacks-source-pvc
Namespace:     default
StorageClass:  gp2
Status:        Pending
Volume:
Labels:        &lt;none&gt;
Annotations:   volume.beta.kubernetes.io/storage-provisioner: ebs.csi.aws.com
               volume.kubernetes.io/selected-node: ip-999-99-99-99.eu-central-1.compute.internal
               volume.kubernetes.io/storage-provisioner: ebs.csi.aws.com
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:    Filesystem
Used By:       affinity-assistant-0b3d266b91-0
               affinity-assistant-53a7c08baf-0
               affinity-assistant-a375f28de3-0
               affinity-assistant-e8cb1a6e15-0
               buildpacks-test-pipeline-run-9rz4l-fetch-repository-pod
Events:
  Type    Reason                Age                     From                         Message
  ----    ------                ----                    ----                         -------
  Normal  ExternalProvisioning  3m43s (x561 over 143m)  persistentvolume-controller  waiting for a volume to be created, either by external provisioner &quot;ebs.csi.aws.com&quot; or manually created by system administrator
</code></pre>
<p>What is this EBS CSI thing and how do we get our cluster working as before?</p>

jonashackt
<p>From EKS 1.23 on <a href="https://kubernetes.io/blog/2019/01/15/container-storage-interface-ga/" rel="nofollow noreferrer">a Container Storage Interface (CSI) driver</a> is needed in order to get your PersisentVolumeClaims served by a PersistentVolume as you are used to from earlier EKS versions.</p>
<p><a href="https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html" rel="nofollow noreferrer">The docs tell us</a>, what needs to be configured:</p>
<h2>Solution: Configure Amazon EBS CSI driver for working PersistentVolumes in EKS</h2>
<p>In essence we need to enable the AWS EBS CSI driver as an EKS addon. But beforehand we need to enable the IAM OIDC provider and create the IAM role for the EBS CSI driver. The easiest way to do both is to use <a href="https://github.com/weaveworks/eksctl" rel="nofollow noreferrer">eksctl</a> (other ways like <a href="https://docs.aws.amazon.com/eks/latest/userguide/managing-ebs-csi.html" rel="nofollow noreferrer">using plain <code>aws</code> cli or the AWS GUI are described in the docs</a>).</p>
<h4>1.) Install eksctl</h4>
<p>We assume here that <a href="https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html" rel="nofollow noreferrer">the aws cli is installed and configured</a> - and you have access to your EKS cluster. To use <code>eksctl</code> we need to install it first. On a Mac use brew like:</p>
<pre><code>brew tap weaveworks/tap
brew install weaveworks/tap/eksctl
</code></pre>
<p>or on Linux use:</p>
<pre><code>curl --silent --location &quot;https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz&quot; | tar xz -C /tmp
sudo mv /tmp/eksctl /usr/local/bin
</code></pre>
<h4>2.) Enable IAM OIDC provider</h4>
<p>A prerequisite for the EBS CSI driver to work is to have an existing AWS Identity and Access Management (IAM) OpenID Connect (OIDC) provider for your cluster. This IAM OIDC provider can be enabled with the following command:</p>
<pre><code>eksctl utils associate-iam-oidc-provider --region=eu-central-1 --cluster=YourClusterNameHere --approve
</code></pre>
<h4>3.) Create Amazon EBS CSI driver IAM role</h4>
<p>Now having <code>eksctl</code> in place, create the IAM role:</p>
<pre><code>eksctl create iamserviceaccount \
  --region eu-central-1 \
  --name ebs-csi-controller-sa \
  --namespace kube-system \
  --cluster YourClusterNameHere \
  --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
  --approve \
  --role-only \
  --role-name AmazonEKS_EBS_CSI_DriverRole
</code></pre>
<p>As you can see AWS maintains a managed policy for us we can simply use (<code>AWS maintains a managed policy, available at ARN arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy</code>). Only if you use encrypted EBS drives <a href="https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/install.md#installation-1" rel="nofollow noreferrer">you need to additionally add configuration to the policy</a>.</p>
<p>The command...</p>
<blockquote>
<p>...deploys an AWS CloudFormation stack that creates an IAM role,
attaches the IAM policy to it, and annotates the existing
ebs-csi-controller-sa service account with the Amazon Resource Name
(ARN) of the IAM role.</p>
</blockquote>
<h4>4.) Add the Amazon EBS CSI add-on</h4>
<p>Now we can finally add the EBS CSI add-on. Therefor we also need the AWS Account id which we can obtain by running <code>aws sts get-caller-identity --query Account --output text</code> (see <a href="https://stackoverflow.com/questions/33791069/quick-way-to-get-aws-account-number-from-the-aws-cli-tools">Quick way to get AWS Account number from the AWS CLI tools?</a>). Now the <code>eksctl create addon</code> command looks like this:</p>
<pre><code>eksctl create addon --name aws-ebs-csi-driver --cluster YourClusterNameHere --service-account-role-arn arn:aws:iam::$(aws sts get-caller-identity --query Account --output text):role/AmazonEKS_EBS_CSI_DriverRole --force
</code></pre>
<p>Now your PersistentVolumeClaim should get the status <code>Bound</code> while a EBS volume got created for you - and the Tekton Pipeline should run again.</p>
