<p>When using emptydir with limits, the pod get killed/recreated when it consumes the disk more than the limits specified. Which is expected behaviour which we also want.</p>
<pre><code>  limits:
    ephemeral-storage: &quot;300Gi&quot;

</code></pre>
<p>However, when using ephemeral from Storage class instead of local, the pod is not getting recreated instead disk usage becomes 100 % and all future build jobs are failing. Is there a way to recreate the pod when such volumes reaches 100 % ?</p>
<pre><code>    - ephemeral:
      name: buiddir
        volumeClaimTemplate:
          metadata:
            labels:
              app.kubernetes.io/name: k8s-helmchart-agent
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              limits:
                storage: 220Gi
              requests:
                storage: 220Gi
            storageClassName: ot-iscsi-basic
            volumeMode: Filesystem
      
</code></pre>

Tamilan
As per <a href="https://stackoverflow.com/questions/71787864/node-not-enough-temp-storage-for-ephemeral-storage">RahulKumarShaw</a>, as explained emptyDir is managed by kubelet on each node. emptyDir: empty at Pod startup, with storage coming locally from the kubelet base directory (usually the root disk) or RAM</p>
&nbsp;</p>
You can also refer this <a href="https://github.com/Azure/AKS/issues/930" rel="nofollow noreferrer">github</a>&nbsp;discussion where different user has reported same kind of issue.</p>
&nbsp;</p>
Conclusion : ephemeral-storage varies from its requirements size because you have file system running on node is using as ephemeral storage. The kubelet also writes node-level container logs into the first file system, and treats these similarly to ephemeral local storage. That might be reason for extending the size of ephemeral storage and as there is also set limit of 300GB for ephemeral storage.</p>
&nbsp;</p>
You can try to <a href="https://www.ibm.com/support/pages/pod-getting-evicted-due-pod-ephemeral-local-storage-usage-exceeds-total-limit-containers-256mi" rel="nofollow noreferrer">increase the ephemeral storage</a>:</p>
<ol>
<li>Make sure&nbsp;wml&nbsp;is in&nbsp;&nbsp;Completed&nbsp;state by running</li>
<blockquote>
<p>oc get wmlbase -n </p>
</blockquote>
<li>Proceed with this step only when step 1 is in&nbsp;Completed&nbsp;state.</li>
<blockquote>
<p>oc get csv -A | grep wml</p>
</blockquote>
<li>Patch or increase the ephemeral storage limit to 512MB by editing the&nbsp;wml&nbsp;csv</li>
<blockquote>
<p>oc edit csv ibm-cpd-wml-operator.v2.2.0 -n ibm-common-services</p>
</blockquote>
<li>And save the csv and now the new&nbsp;wml&nbsp;operator&nbsp;pod comes up with increased ephemeral storage under limits.</li>
</ol>
