<p>I have Kubernetes cluster running on a VM. A truncated overview of the mounts is:</p>
<pre class="lang-sh prettyprint-override"><code>$ df -h
Filesystem                    Size  Used Avail Use% Mounted on
/dev/sda1                      20G  4.5G   15G  24% /
/dev/mapper/vg001-lv--docker  140G   33G  108G  23% /var/lib/docker
</code></pre>
<p>As you can see, I added an extra disk to store the docker images and its volumes. However, when querying the node's capacity, the following is returned</p>
<pre><code>Capacity:
  cpu:                12
  ephemeral-storage:  20145724Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             65831264Ki
  nvidia.com/gpu:     1
  pods:               110
</code></pre>
<p><code>ephemeral-storage</code> is <code>20145724Ki</code> which is 20G, referring to the disk mounted at <code>/</code>.</p>
<p>How does Kubelet calculate its <code>ephemeral-storage</code>? Is it simply looking at the disk space available at <code>/</code>? Or is it looking at another folder like <code>/var/log/containers</code>?</p>
<p><a href="https://stackoverflow.com/questions/58269443/how-can-we-increase-the-size-of-ephemeral-storage-in-a-kubernetes-worker-node">This is a similar post</a> where the user eventually succumbed to increasing the disk mounted at <code>/</code>.</p>

<p><strong>Some theory</strong></p>
<p>By default <code>Capacity</code> and <code>Allocatable</code> for ephemeral-storage in standard kubernetes environment is sourced from filesystem (mounted to /var/lib/kubelet).
This is the default location for kubelet directory.</p>
<p><a href="https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/#eviction-signals" rel="noreferrer">The kubelet supports the following filesystem partitions:</a></p>
<blockquote>
<ol>
<li><code>nodefs</code>: The node's main filesystem, used for local disk volumes, emptyDir, log storage, and more. For example,  <code>nodefs</code>  contains
<code>/var/lib/kubelet/</code>.</li>
<li><code>imagefs</code>: An optional filesystem that container runtimes use to store container images and container writable layers.</li>
</ol>
<p>Kubelet auto-discovers these filesystems and ignores other
filesystems. Kubelet does not support other configurations.</p>
</blockquote>
<p>From <a href="https://kubernetes.io/docs/concepts/storage/volumes/#resources" rel="noreferrer">Kubernetes website</a> about volumes:</p>
<blockquote>
<p>The storage media (such as Disk or SSD) of an <code>emptyDir</code> volume is
determined by the medium of the filesystem holding the kubelet root
dir (typically <code>/var/lib/kubelet</code>).</p>
</blockquote>
<p>Location for kubelet directory can be configured by providing:</p>
<ol>
<li><a href="https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/" rel="noreferrer">Command line parameter during kubelet initialization</a></li>
</ol>
<p><code>--root-dir</code> string
Default: /var/lib/kubelet</p>
<ol start="2">
<li>Via <a href="https://kubernetes.io/docs/reference/config-api/kubeadm-config.v1beta3/#kubeadm-init-configuration-types" rel="noreferrer">kubeadm with config file</a> (e.g.)</li>
</ol>
<pre><code>apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
nodeRegistration:
  kubeletExtraArgs:
    root-dir: &quot;/data/var/lib/kubelet&quot;
</code></pre>
<p><a href="https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/control-plane-flags/#customizing-the-kubelet" rel="noreferrer">Customizing kubelet</a>:</p>
<blockquote>
<p>To customize the kubelet you can add a <code>KubeletConfiguration</code> next to
the <code>ClusterConfiguration</code> or <code>InitConfiguration</code> separated by <code>---</code>
within the same configuration file. This file can then be passed to
<code>kubeadm init</code>.</p>
</blockquote>
<p>When bootstrapping kubernetes cluster using kubeadm, <code>Capacity</code> reported by <code>kubectl get node</code> is equal to the disk capacity mounted into <code>/var/lib/kubelet</code></p>
<p>However <code>Allocatable</code> will be reported as:
<code>Allocatable</code> = <code>Capacity</code> - <code>10% nodefs</code> using the standard kubeadm configuration, since <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/#hard-eviction-thresholds" rel="noreferrer">the kubelet has the following default hard eviction thresholds:</a></p>
<ul>
<li><code>nodefs.available&lt;10%</code></li>
</ul>
<p>It can be configured during <a href="https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/" rel="noreferrer">kubelet initialization</a> with:
<code>-eviction-hard</code> mapStringString
Default: imagefs.available&lt;15%,memory.available&lt;100Mi,nodefs.available&lt;10%</p>
<hr />
<p><strong>Example</strong></p>
<p>I set up a test environment for Kubernetes with a master node and two worker nodes (worker-1 and worker-2).</p>
<p>Both worker nodes have volumes of the same capacity: 50Gb.</p>
<p>Additionally, I mounted a second volume with a capacity of 20Gb  for the Worker-1 node at the path  <code>/var/lib/kubelet</code>.
Then I created a cluster with kubeadm.</p>
<p><strong>Result</strong></p>
<p><em>From worker-1 node:</em></p>
<pre><code>skorkin@worker-1:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        49G  2.8G   46G   6% /
...
/dev/sdb         20G   45M   20G   1% /var/lib/kubelet
</code></pre>
<p>and</p>
<pre><code>Capacity:
  cpu:                2
  ephemeral-storage:  20511312Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             4027428Ki
  pods:               110
</code></pre>
<p>Size of ephemeral-storage is the same as volume mounted at /var/lib/kubelet.</p>
<p><em>From worker-2 node:</em></p>
<pre><code>skorkin@worker-2:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        49G  2.7G   46G   6% /
</code></pre>
<p>and</p>
<pre><code>Capacity:
  cpu:                2
  ephemeral-storage:  50633164Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             4027420Ki
  pods:               110  
</code></pre>
