<p>We recently upgraded our EKS environment to v1.12.7. After the upgrade we noticed that there is now an "allocatable" resource called <code>attachable-volumes-aws-ebs</code> and in our environment we have many EBS volumes attached to each node (they were all generated dynamically via PVCs).</p>

<p>Yet on every node in the "allocated resources" section, it shows 0 volumes attached:</p>

<pre><code>Allocatable:
 attachable-volumes-aws-ebs:  25
 cpu:                         16
 ephemeral-storage:           96625420948
 hugepages-1Gi:               0
 hugepages-2Mi:               0
 memory:                      64358968Ki
 pods:                        234
...
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource                    Requests           Limits
  --------                    --------           ------
  cpu                         5430m (33%)        5200m (32%)
  memory                      19208241152 (29%)  21358Mi (33%)
  attachable-volumes-aws-ebs  0                  0
</code></pre>

<p>Because of this, the scheduler is continuing to try and attach new volumes to nodes that already have 25 volumes attached.</p>

<p>How do we get kubernetes to recognize the volumes that are attached so that the scheduler can act accordingly?</p>

<p>First check your pods status, they can have <strong>pending</strong> status that's why probably you volumes does't count. Your volume could stuck in attaching state when using multiple <strong>PersistentVolumeClaim</strong> too.</p>

<p>Your volumes may be not attached due to <strong>NodeWithImpairedVolumes=true:NoSchedule</strong> flag and at the same time number of your <strong>attachable-volumes-aws-ebs</strong>.</p>

<p>Try to execute: </p>

<p><code>$ kubectl taint nodes node1 key:NoSchedule-</code> </p>

<p>on every node with this label (NodeWithImpairedVolumes=true:NoSchedule).</p>

<p>If you use <strong>awsElasticBlockStore</strong> , there are some restrictions when using an awsElasticBlockStore volume:</p>

<ul>
<li>nodes on which Pods are running must be AWS EC2 instances
those instances need to be in the same region and availability-zone as the EBS volume
EBS only supports a single EC2 instance mounting a volume.</li>
</ul>

<p>You can use <strong>count/*</strong> resource quota, an object is charged against the quota if it exists in server storage. These types of quotas are useful to protect against exhaustion of storage resources use <strong>count/persistentvolumeclaims</strong>.</p>

<p>Allocatable will be computed by the Kubelet and reported to the API server. It is defined to be:</p>

<p><strong>[Allocatable] = [Node Capacity] - [Kube-Reserved] - [System-Reserved] - [Hard-Eviction-Threshold]</strong></p>

<p>Note: Since kernel usage can fluctuate and is out of kubernetes control, it will be reported as a separate value (probably via the metrics API). Reporting kernel usage is out-of-scope for this proposal.</p>
