<p>I specified ephemeral storage limits and requests on a container, and I’m using latest amazon-eks-node-1.12-v20190701 image.</p>

<p>My issue is that when I specify <code>ephemeral-storage: 1Mi</code> my pod is being evicted for using too much. I can easily add more but I would like to know how exactly this works and kubernetes documentation is not very helpful.</p>

<p>From what I read, there is a Garbage Collection <code>that will clean up unused images and unused containers. Kubelet will perform garbage collection for containers every minute and garbage collection for images every five minutes.</code></p>

<p>Is that apply to ephemeral storage too?</p>

<p>How do I check what’s inside the ephemeral storage (via container or a node?) and how can I reduce it?</p>

<p>Is it true that when ephemeral-storage is up to 85% and when garbage collection run, it will be cleaned up?</p>

<p>Sorry for a lot of questions, just trying to understand and it’s difficult to find a simple answe</p>

tr53
<p>As far as I'm aware (please correct me if this is wrong) there is no Garbage Collection for Ephemeral Storage.</p>

<p>As we can read in the Kubernetes documentation regarding <a href="https://kubernetes.io/docs/concepts/workloads/controllers/garbage-collection/" rel="nofollow noreferrer">Garbage Collection</a>.</p>

<blockquote>
  <p>Some Kubernetes objects are owners of other objects. For example, a ReplicaSet is the owner of a set of Pods. The owned objects are called  <em>dependents</em>  of the owner object. Every dependent object has a  <code>metadata.ownerReferences</code>  field that points to the owning object.</p>
  
  <p>Sometimes, Kubernetes sets the value of  <code>ownerReference</code>  automatically. For example, when you create a ReplicaSet, Kubernetes automatically sets the  <code>ownerReference</code>  field of each Pod in the ReplicaSet. In 1.8, Kubernetes automatically sets the value of  <code>ownerReference</code>  for objects created or adopted by ReplicationController, ReplicaSet, StatefulSet, DaemonSet, Deployment, Job and CronJob.</p>
  
  <p>You can also specify relationships between owners and dependents by manually setting the  <code>ownerReference</code>  field.</p>
</blockquote>

<p>So we can take advantage of Garbage Collection to delete dependent objects.</p>

<blockquote>
  <p>When you delete an object, you can specify whether the object’s dependents are also deleted automatically. Deleting dependents automatically is called  <em>cascading deletion</em>. There are two modes of  <em>cascading deletion</em>:  <em>background</em>  and  <em>foreground</em>.</p>
  
  <p>If you delete an object without deleting its dependents automatically, the dependents are said to be  <em>orphaned</em>.</p>
</blockquote>

<p>In Kubernetes v1.16 new resource was introduced <a href="https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#local-ephemeral-storage" rel="nofollow noreferrer">Local ephemeral storage</a>
></p>

<blockquote>
  <p>In each Kubernetes node, kubelet’s root directory (/var/lib/kubelet by default) and log directory (/var/log) are stored on the root partition of the node. This partition is also shared and consumed by Pods via emptyDir volumes, container logs, image layers and container writable layers.</p>
</blockquote>

<p>And to answer your question </p>

<blockquote>
  <p>How do I check what’s inside the ephemeral storage (via container or a node?) and how can I reduce it?</p>
</blockquote>

<p>It might be possible to exec into the docker container that have the storage mounted and see whats there, but other then that I don't see a way of accessing the content.
As for reducing the storage (I did not tested that) it might remove the content of the ephemeral storage because it's not extendable.</p>
