<p><strong>Problem:</strong></p>
<p>I upgraded from AWS EKS cluster to v1.23 from v1.22 and all of a sudden, all the pods that had Persistent Volume Claim (PVC) and Persistent Volume (PV) started failing with the errors like <code>FailedAttachVolume AttachVolume.Attach failed</code> and <code>FailedMount MountVolume.WaitForAttach failed</code> for the AWS EBS volumes.</p>
<p>Pods were giving the following error: <code>Unable to attach or mount volumes timed out waiting for the condition</code></p>
<p><strong>Solutions Tried:</strong></p>
<ul>
<li>I tried adding AWS EBS CSI Driver add-on in the AWS EKS cluster but still it didn't work.</li>
<li>I tried removing the annotation for migration to this new provisioner on PVCs but that didn't work either.</li>
<li>I also tried adding storage class for <code>gp3</code> AWS EBS volume type with AWS EBS CSI Driver as the new provisioner as I was using <code>gp2</code> but that didn't work either.</li>
</ul>
<p><strong>Note:</strong> AWS EBS volumes were of type <code>gp2</code>.</p>

Abdullah Khawer
<p>After adding <code>AmazonEBSCSIDriverPolicy</code> AWS IAM policy to the AWS IAM role that is attached to all the AWS EKS nodes (AWS EC2 instances) and then adding the <code>AWS EBS CSI Driver</code> add-on in the AWS EKS cluster, errors were resolved and PVC got attached successfully. I don't see any issues related to Persistent Volume Claims (PVC), Persistent Volumes (PV), AWS EBS volumes, and pods anymore.</p>
<p><strong>Note:</strong> I already had an AWS IAM OpenID Connect (OIDC) provider for my AWS EKS cluster which is a prerequisite for this. In your case, there could be some other issue and the resolution steps might differ so please check the referred document.</p>
<p>Reference: <a href="https://repost.aws/knowledge-center/eks-troubleshoot-ebs-volume-mounts" rel="nofollow noreferrer">How do I troubleshoot issues with my EBS volume mounts in Amazon EKS?</a></p>
