<p>So far our on-prem kubernetes cluster is working fine. Lately we are seeing jobs are  failing with the below error. I checked there are no space issues on Kube master as well as the worker nodes. There is plenty of space available under "/var/lib" as well as under persistent volume claims. </p>

<pre><code>Version:

Client Version: v1.17.2
Server Version: v1.17.2

Host OS:
Centos 7.7

CNI:
Weave
</code></pre>

<p>Error:</p>

<blockquote>
  <p>The node was low on resource: ephemeral-storage.Container main was
  using 5056Ki, which exceeds its request of 0. Container wait was using
  12Ki, which exceeds its request of 0.</p>
</blockquote>

<p>any pointers will be helpful.</p>

<p>Thanks,
CS </p>

<p>The main reason why this could be happening is that pod logs, or <code>emptyDir</code> usage are filling up your ephemeral storage. </p>

<blockquote>
  <p>Docker takes a conservative approach to cleaning up unused objects (often referred to as “garbage collection”), such as images, containers, volumes, and networks: these objects are generally not removed unless you explicitly ask Docker to do so. This can cause Docker to use extra disk space.</p>
</blockquote>

<p>You can use docker function called <code>prune</code>. This will clean up the system from unused objects. If you wish to cleanup multiple objects you can use <code>docker system prune</code>. Check here more about <a href="https://docs.docker.com/config/pruning/" rel="nofollow noreferrer">prunning</a>.</p>

<p>There is also another tool called <code>Garbage collector</code>. It`s docker tool that removes.  unused/abandoned/orphaned blobs. Check here <a href="https://docs.docker.com/registry/garbage-collection/" rel="nofollow noreferrer">more</a> about it. </p>

<blockquote>
  <p>In the context of the Docker registry, garbage collection is the process of removing blobs from the filesystem when they are no longer referenced by a manifest. Blobs can include both layers and manifests.</p>
</blockquote>

<p>If this does`t help you can try to <a href="https://docs.docker.com/config/containers/logging/configure/" rel="nofollow noreferrer">configure logging driver</a> and set its limit: </p>

<pre><code>{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3",
    "labels": "production_status",
    "env": "os,customer"
  }
}
</code></pre>

<p>There is also another option if <code>emptyDir</code> has been used. Using <code>emptyDir</code> you allow 
container to write any amount of storage to it's node fs.  You can <a href="https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#requests-and-limits-setting-for-local-ephemeral-storage" rel="nofollow noreferrer">request or limit</a> settings for local ephemeral storage by setting up: </p>

<pre class="lang-yaml prettyprint-override"><code>spec:
  containers:
  - name: test
    image: test-image
    resources:
      requests:
        ephemeral-storage: "1Gi"
      limits:
        ephemeral-storage: "1Gi"
  - name: test
    image: test-image2
    resources:
      requests:
        ephemeral-storage: "2Gi"
      limits:
        ephemeral-storage: "2Gi"
</code></pre>

<p>You can also check the containers running using <code>docker ps</code> and then inspect the container by yourself and locate the fs. </p>

<p>It should be found at this location: </p>

<pre><code>/var/lib/docker/containers/&lt;container-id&gt;/&lt;container-id&gt;-json.log
</code></pre>

<p>Let me know if that helps. </p>
