<p>Currently I am trying to have volume persistence for my MYSQL database using Kubernetes with Kubeadm.
The environment is based on an amazon EC2 instance using EBS storage disks.</p>

<p>As you can see below a storage class, a persistent volume as well as a persistent volume claim have been implemented in order to have a mysql persistence.
However an error occurs when I try to deploy the mysql pod (on the attached image).</p>

<p><strong>mysql-pv.yml:</strong></p>

<pre><code>kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: standard
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2
reclaimPolicy: Retain
mountOptions:
  - debug
volumeBindingMode: Immediate
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: mysql-pv
  labels:
    type: amazonEBS
spec:
  capacity:
    storage: 5Gi
  storageClassName: standard
  accessModes:
    - ReadWriteOnce
  awsElasticBlockStore:
    volumeID: vol-ID
    fsType: ext4
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mysql-pv-claim
spec:
  storageClassName: standard
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
</code></pre>

<p><strong>mysql.yml</strong>:</p>

<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql
spec:
  selector:
    matchLabels:
      app: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
        - image: mysql:5.7.30
          name: mysql
          env:
            - name: MYSQL_ROOT_PASSWORD
              value: MYPASSWORD
          ports:
            - containerPort: 3306
              name: mysql
          volumeMounts:
            - name: mysql-persistent-storage
              mountPath: /var/lib/mysql
      volumes:
        - name: mysql-persistent-storage
          persistentVolumeClaim:
            claimName: mysql-pv-claim
---
apiVersion: v1
kind: Service
metadata:
  name: mysql
spec:
  type: NodePort
  ports:
    - port: 3306
      targetPort: 3306
      nodePort: 31306
  selector:
    app: mysql
</code></pre>

<p>This is my mysql pod description:</p>

<pre><code>Name:           mysql-5c9788fc65-jq2nh
Namespace:      default
Priority:       0
Node:           ip-172-31-31-210/172.31.31.210
Start Time:     Sat, 23 May 2020 12:19:24 +0000
Labels:         app=mysql
                pod-template-hash=5c9788fc65
Annotations:    &lt;none&gt;
Status:         Pending
IP:             
IPs:            &lt;none&gt;
Controlled By:  ReplicaSet/mysql-5c9788fc65
Containers:
  mysql:
    Container ID:   
    Image:          mysql:5.7.30
    Image ID:       
    Port:           3306/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:
      MYSQL_ROOT_PASSWORD:  MYPASS
    Mounts:
      /data/ from mysql-persistent-storage (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-cshk2 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  mysql-persistent-storage:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  mysql-pv-claim
    ReadOnly:   false
  default-token-cshk2:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-cshk2
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  &lt;none&gt;
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
</code></pre>

<p>Here is the error I get:</p>

<pre><code>Events:
Type     Reason       Age        From                       Message
----     ------       ----       ----                       -------
Normal   Scheduled    &lt;unknown&gt;  default-scheduler          
Successfully assigned default/mysql-5c9788fc65-jq2nh to ip-172-31-31-210
Warning  FailedMount  39m        kubelet, ip-172-31-31-210  MountVolume.SetUp failed for volume "mysql-pv" : mount failed: exit status 32

Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/29d5cee7-da11-4a0c-b5aa-e262f919d1ba/volumes/kubernetes.io~aws-ebs/mysql-pv --scope -- mount  -o bind /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/vol-06212746d87534157 /var/lib/kubelet/pods/29d5cee7-da11-4a0c-b5aa-e262f919d1ba/volumes/kubernetes.io~aws-ebs/mysql-pv
Output: Running scope as unit: run-r11fefbbda1d241c2985931d3adaaa969.scope
mount: /var/lib/kubelet/pods/29d5cee7-da11-4a0c-b5aa-e262f919d1ba/volumes/kubernetes.io~aws-ebs/mysql-pv: special device /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/vol-06212746d87534157 does not exist.
Warning  FailedMount  39m  kubelet, ip-172-31-31-210  MountVolume.SetUp failed for volume "mysql-pv" : mount failed: exit status 32
</code></pre>

<p>Someone can help me ?</p>

Alexandre FERREIRA
<p>Check for the state of PV and PVC if the PVC is in bounded state or not.</p>

<p><code>kubectl describe pvc mysql-pv-claim</code>
<code>kubectl describe pv mysql-pv</code>
Do you have the <a href="https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html" rel="nofollow noreferrer">EBS CSI driver</a> installed?</p>

<p>Other reason can be, I think you missed adding the option of <strong>--cloud-provider=aws</strong>, this is required by CCM for the nodes. Check out the similar <a href="https://github.com/kubernetes/kubernetes/issues/70921" rel="nofollow noreferrer">issue</a>.</p>

<p>The following link has all the IAM permissions and a working example on how to create and mount an EBS volume in Kubernetes from docs on <a href="https://docs.docker.com/ee/ucp/kubernetes/storage/configure-aws-storage/#cluster-configuration" rel="nofollow noreferrer">cluster configuration</a> for using EBS. </p>

<p>With a kubeadm configuration, configuration is defined in: <strong>/var/lib/kubelet/config.yaml</strong> and <strong>/var/lib/kubelet/kubeadm-flags.env</strong></p>

<p>If the cluster was deployed using <strong>kubeadm</strong> then, define environment variable on all nodes in <code>kubeadm-flags.env</code>
To resolve this manually you need to add the <a href="https://github.com/kubernetes/cloud-provider-aws#readme" rel="nofollow noreferrer">--cloud-provider=aws</a> tag to the <code>kubeadm-flags.env</code> and restarted the services, which will resolve the issue: </p>

<p><code>systemctl daemon-reload &amp;&amp; systemctl restart kubelet</code>
or provide the following <a href="https://kubernetes.io/docs/concepts/cluster-administration/cloud-providers/#kubeadm" rel="nofollow noreferrer">configuration</a> for kubeadm. Change openstack to AWS in your case.</p>

<p>Check the following <a href="https://www.nebulaworks.com/blog/2019/08/27/leveraging-aws-ebs-for-kubernetes-persistent-volumes/" rel="nofollow noreferrer">blog</a> for better understanding.</p>

<pre><code>kind: InitConfiguration
nodeRegistration:
  kubeletExtraArgs:
    cloud-provider: "aws"
    cloud-config: "/etc/kubernetes/cloud.conf"
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
apiServer:
  extraArgs:
    cloud-provider: "aws"
    #cloud-config: "/etc/kubernetes/cloud.conf"
  extraVolumes:
  - name: cloud
    hostPath: "/etc/kubernetes/cloud.conf"
    mountPath: "/etc/kubernetes/cloud.conf"
controllerManager:
  extraArgs:
    cloud-provider: "aws"
    #cloud-config: "/etc/kubernetes/cloud.conf"
  extraVolumes:
  - name: cloud
    hostPath: "/etc/kubernetes/cloud.conf"
    mountPath: "/etc/kubernetes/cloud.conf"```

</code></pre>
