title: Unable to attach aws ebs volume, error "instance not found"
body: <p>Info:</p>
  <ul>
  <li>Kubernetes Server version: 1.14</li>
  <li>AWS Cloud Provider</li>
  <li>EBS volume, storageclass</li>
  </ul>
  <p>Details:
  I have installed statefulset in our kubernetes cluster, however, it stuck it &quot;ContainerCreating&quot; status. Upon checking the logs, the error is &quot;AttachVolume.Attach failed for volume pvc-xxxxxx: error finding instance ip-xxxxx : &quot;instance not found&quot;</p>
  <p>It was succesfully installed around 17 days ago, but re-installing for an update caused the pod to stuck in ContainerCreating.</p>
  <p>Manual attaching volume to the instance works. But doing it via storage class is not working and stuck in ContainerCreating status.</p>
  <p>storageclass:</p>
  <pre><code>apiVersion: storage.k8s.io/v1
  kind: StorageClass
  metadata:
    annotations:
      storageclass.kubernetes.io/is-default-class: &quot;true&quot;
    name: ssd-default
  allowVolumeExpansion: true
  parameters:
    encrypted: &quot;true&quot;
    type: gp2
  provisioner: kubernetes.io/aws-ebs
  reclaimPolicy: Delete
  volumeBindingMode: Immediate
  </code></pre>
  <p>pvc yaml:</p>
  <pre><code>apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    annotations:
      pv.kubernetes.io/bind-completed: &quot;yes&quot;
      pv.kubernetes.io/bound-by-controller: &quot;yes&quot;
      volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/aws-ebs
    finalizers:
    - kubernetes.io/pvc-protection
    labels:
      app.kubernetes.io/instance: thanos-store
      app.kubernetes.io/name: thanos-store
    name: data-thanos-store-0
    namespace: thanos
  spec:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 3Gi
    storageClassName: ssd-default
    volumeMode: Filesystem
    volumeName: pvc-xxxxxx
  status:
    accessModes:
    - ReadWriteOnce
    capacity:
      storage: 3Gi
    phase: Bound
  </code></pre>
  <p>pv yaml:</p>
  <pre><code>apiVersion: v1
  kind: PersistentVolume
  metadata:
    annotations:
      kubernetes.io/createdby: aws-ebs-dynamic-provisioner
      pv.kubernetes.io/bound-by-controller: &quot;yes&quot;
      pv.kubernetes.io/provisioned-by: kubernetes.io/aws-ebs
    finalizers:
    - kubernetes.io/pv-protection
    labels:
      failure-domain.beta.kubernetes.io/region: ap-xxx
      failure-domain.beta.kubernetes.io/zone: ap-xxx
    name: pvc-xxxx
  spec:
    accessModes:
    - ReadWriteOnce
    awsElasticBlockStore:
      fsType: ext4
      volumeID: aws://xxxxx
    capacity:
      storage: 3Gi
    claimRef:
      apiVersion: v1
      kind: PersistentVolumeClaim
      name: data-athena-thanos-store-0
      namespace: thanos
    nodeAffinity:
      required:
        nodeSelectorTerms:
        - matchExpressions:
          - key: failure-domain.beta.kubernetes.io/region
            operator: In
            values:
            - ap-xxx
          - key: failure-domain.beta.kubernetes.io/zone
            operator: In
            values:
            - ap-xxx
    persistentVolumeReclaimPolicy: Delete
    storageClassName: ssd-default
    volumeMode: Filesystem
  status:
    phase: Bound
  </code></pre>
  <p>Describe pvc:</p>
  <pre><code>Name:          data-athena-thanos-store-0
  Namespace:     athena-thanos
  StorageClass:  ssd-encrypted
  Status:        Bound
  Volume:        pvc-xxxx
  Labels:        app.kubernetes.io/instance=athena-thanos-store
                 app.kubernetes.io/name=athena-thanos-store
  Annotations:   pv.kubernetes.io/bind-completed: yes
                 pv.kubernetes.io/bound-by-controller: yes
                 volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/aws-ebs
  Finalizers:    [kubernetes.io/pvc-protection]
  Capacity:      3Gi
  Access Modes:  RWO
  VolumeMode:    Filesystem
  Mounted By:    athena-thanos-store-0
  </code></pre>