<p>I am trying to deploy a stateful set mounted on a <a href="https://kubernetes.io/docs/concepts/storage/persistent-volumes/" rel="nofollow noreferrer">Persistent Volume</a>.</p>

<p>I installed Kubernetes on AWS via <a href="https://github.com/kubernetes/kops/blob/master/docs/aws.md" rel="nofollow noreferrer">kops</a>.</p>

<pre><code>$ kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"clean", BuildDate:"2018-02-07T12:22:21Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"clean", BuildDate:"2018-02-07T11:55:20Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
</code></pre>

<p>According to <a href="https://github.com/kubernetes/kubernetes/issues/28962" rel="nofollow noreferrer">this issue</a> I need to create the PVC first:</p>

<pre><code>kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: zk-data-claim
spec:
  storageClassName: default
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: zk-logs-claim
spec:
  storageClassName: default
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
</code></pre>

<p>The <code>default</code> storage class exists, and the PVC binds a PV successfully:</p>

<pre><code>$ kubectl get sc
NAME            PROVISIONER             AGE
default         kubernetes.io/aws-ebs   20d
gp2 (default)   kubernetes.io/aws-ebs   20d
ssd (default)   kubernetes.io/aws-ebs   20d

$ kubectl get pvc
NAME            STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
zk-data-claim   Bound     pvc-5584fdf7-3853-11e8-a73b-02bb35448afe   2Gi        RWO            default        11m
zk-logs-claim   Bound     pvc-5593e249-3853-11e8-a73b-02bb35448afe   2Gi        RWO            default        11m
</code></pre>

<p>I can see these two volumes in the EC2 EBS Volumes list as "available" at first, but then later becomes "in-use".  </p>

<p>And then ingest it in my StatefulSet</p>

<pre><code>apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: zk
spec:
  serviceName: zk-cluster
  replicas: 3
  template:
    metadata:
      labels:
        app: zookeeper
    spec:
      volumes:
        - name: zk-data
          persistentVolumeClaim:
            claimName: zk-data-claim
        - name: zk-logs
          persistentVolumeClaim:
            claimName: zk-logs-claim

      containers:
      ....
        volumeMounts:
        - name: zk-data
          mountPath: /opt/zookeeper/data
        - name: zk-logs
          mountPath: /opt/zookeeper/logs
</code></pre>

<p><strong>Which fails with</strong> </p>

<pre><code>Unable to mount volumes for pod "zk-0_default(83b8dc93-3850-11e8-a73b-02bb35448afe)": timeout expired waiting for volumes to attach/mount for pod "default"/"zk-0". list of unattached/unmounted volumes=[zk-data zk-logs]
</code></pre>

<p>I'm working in the default namespace.</p>

<p>Any ideas what could be causing this failure?</p>

<p>The problem was that my cluster was made with C5 nodes.  C5 and M5 nodes follow a different naming convention (<a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/nvme-ebs-volumes.html" rel="nofollow noreferrer">NVMe</a>), and the naming is not recognised.</p>

<p>Recreate the cluster with t2 type nodes.</p>
