<p>I've been runnning into what should be a simple issue with my airflow scheduler. Every couple of weeks, the scheduler becomes <code>Evicted</code>. When I run a describe on the pod, the issue is because <code>The node was low on resource: ephemeral-storage. Container scheduler was using 14386916Ki, which exceeds its request of 0.</code></p>
<p>The question is two fold. First, why is the scheduler utilizing ephemeral-storage? And second, is it possible to do add ephemeral-storage when running on eks?</p>
<p>Thanks!</p>

JaMo
<p>I believe Ephemeral Storage is not Airflow's question but more of the configuration of your K8S cluster.</p>
<p>Assuming we are talking about OpenShift' ephemeral storage:</p>
<p><a href="https://docs.openshift.com/container-platform/4.9/storage/understanding-ephemeral-storage.html" rel="nofollow noreferrer">https://docs.openshift.com/container-platform/4.9/storage/understanding-ephemeral-storage.html</a></p>
<p>This can be configured in your cluster and it wil make &quot;/var/log&quot; ephemeral.</p>
<p>I think the problem is that <code>/var/logs</code> gets full. Possibly some of the system logs (not from airlfow but from some other processes running in the same container). I think a solution will be to have a job that cleans that system log periodically.</p>
<p>For example we have this script that cleans-up Airlfow logs:</p>
<p><a href="https://github.com/apache/airflow/blob/main/scripts/in_container/prod/clean-logs.sh" rel="nofollow noreferrer">https://github.com/apache/airflow/blob/main/scripts/in_container/prod/clean-logs.sh</a></p>
