<p>Info:</p>
<ul>
<li>Kubernetes Server version: 1.14</li>
<li>AWS Cloud Provider</li>
<li>EBS volume, storageclass</li>
</ul>
<p>Details:
I have installed statefulset in our kubernetes cluster, however, it stuck it &quot;ContainerCreating&quot; status. Upon checking the logs, the error is &quot;AttachVolume.Attach failed for volume pvc-xxxxxx: error finding instance ip-xxxxx : &quot;instance not found&quot;</p>
<p>It was succesfully installed around 17 days ago, but re-installing for an update caused the pod to stuck in ContainerCreating.</p>
<p>Manual attaching volume to the instance works. But doing it via storage class is not working and stuck in ContainerCreating status.</p>
<p>storageclass:</p>
<pre><code>apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: &quot;true&quot;
  name: ssd-default
allowVolumeExpansion: true
parameters:
  encrypted: &quot;true&quot;
  type: gp2
provisioner: kubernetes.io/aws-ebs
reclaimPolicy: Delete
volumeBindingMode: Immediate
</code></pre>
<p>pvc yaml:</p>
<pre><code>apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: &quot;yes&quot;
    pv.kubernetes.io/bound-by-controller: &quot;yes&quot;
    volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/aws-ebs
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app.kubernetes.io/instance: thanos-store
    app.kubernetes.io/name: thanos-store
  name: data-thanos-store-0
  namespace: thanos
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 3Gi
  storageClassName: ssd-default
  volumeMode: Filesystem
  volumeName: pvc-xxxxxx
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 3Gi
  phase: Bound
</code></pre>
<p>pv yaml:</p>
<pre><code>apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    kubernetes.io/createdby: aws-ebs-dynamic-provisioner
    pv.kubernetes.io/bound-by-controller: &quot;yes&quot;
    pv.kubernetes.io/provisioned-by: kubernetes.io/aws-ebs
  finalizers:
  - kubernetes.io/pv-protection
  labels:
    failure-domain.beta.kubernetes.io/region: ap-xxx
    failure-domain.beta.kubernetes.io/zone: ap-xxx
  name: pvc-xxxx
spec:
  accessModes:
  - ReadWriteOnce
  awsElasticBlockStore:
    fsType: ext4
    volumeID: aws://xxxxx
  capacity:
    storage: 3Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: data-athena-thanos-store-0
    namespace: thanos
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: failure-domain.beta.kubernetes.io/region
          operator: In
          values:
          - ap-xxx
        - key: failure-domain.beta.kubernetes.io/zone
          operator: In
          values:
          - ap-xxx
  persistentVolumeReclaimPolicy: Delete
  storageClassName: ssd-default
  volumeMode: Filesystem
status:
  phase: Bound
</code></pre>
<p>Describe pvc:</p>
<pre><code>Name:          data-athena-thanos-store-0
Namespace:     athena-thanos
StorageClass:  ssd-encrypted
Status:        Bound
Volume:        pvc-xxxx
Labels:        app.kubernetes.io/instance=athena-thanos-store
               app.kubernetes.io/name=athena-thanos-store
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/aws-ebs
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      3Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Mounted By:    athena-thanos-store-0
</code></pre>

Yejin
<p>The <code>FailedAttachVolume</code> error occurs when an EBS volume can’t be detached from an instance and thus cannot be attached to another. The EBS volume has to be in the available state to be attached. <code>FailedAttachVolume</code> is usually a symptom of an underlying failure to unmount and detach the volume.</p>
<p>Notice that while describing the PVC the <code>StorageClass</code> name is <code>ssd-encrypted</code> which is a mismatch with the config you showed earlier where the <code>kind: StorageClass</code> name is <code>ssd-default</code>. That's why you can mount the volume manually but not via the <code>StorageClass</code>. You can drop and recreate the <code>StorageClass</code> with a proper data.</p>
<p>Also, I recommend going through <a href="https://kubernetes.io/blog/2018/10/11/topology-aware-volume-provisioning-in-kubernetes/" rel="nofollow noreferrer">this article</a> and using <code>volumeBindingMode: WaitForFirstConsumer</code> instead of <code>volumeBindingMode: Immediate</code>. This setting instructs the volume provisioner to not create a volume immediately, and instead, wait for a pod using an associated PVC to run through scheduling.</p>
