<p>I want to re-partition a dataset and then write it on the destination path. However, my pods are getting evicted due to <code>DiskPressure</code>. Spark only shows that it lost a worker but when I see the <code>events</code> in my OpenShift console, I see the that the pod(worker) was evicted.</p>
<p>Here is how I am re-partitioning:</p>
<pre><code>df = df.repartition(&quot;created_year&quot;, &quot;created_month&quot;, &quot;created_day&quot;)
df.write.partitionBy(&quot;created_year&quot;, &quot;created_month&quot;, &quot;created_day&quot;).mode(&quot;overwrite&quot;).parquet(dest_path)
</code></pre>
<p>There are around 38k partitions:</p>
<pre><code>Job Id  ▾
Description
Submitted
Duration
Stages: Succeeded/Total Tasks (for all stages): Succeeded/Total
1   
parquet at NativeMethodAccessorImpl.java:0
(kill)parquet at NativeMethodAccessorImpl.java:0    2020/08/11 21:35:46 1.5 h   0/2 
2166/38281 (5633 failed)
</code></pre>
<p>Spark configurations are as follows:</p>
<pre><code>def create_spark_config(spark_cluster, executor_memory='16g', executor_cores='4', max_cores='16'):
    print('Spark cluster is: {}'.format(spark_cluster))
    sc_conf = (
        pyspark.SparkConf().setMaster(spark_cluster) \
        .set('spark.driver.host', HOSTNAME) \
        .set('spark.driver.port', 42000) \
        .set('spark.driver.bindAddress', '0.0.0.0') \
        .set('spark.driver.blockManager.port', 42100) \
        .set('spark.executor.memory', '5G') \
        .set('spark.driver.memory', '3G') \
        .set('spark.sql.parquet.enableVectorizedReader', True) \
        .set('spark.sql.files.ignoreCorruptFiles', True)
    )
    return sc_conf
</code></pre>
<p>I am not able to figure out what is causing the DiskPressure, and how can I stop it?</p>
<p>I read some answers and articles about DiskPressure and its handling but they were more generic and not pertaining to Spark.</p>
<p>Spark has 6 workers, each with 5GB of memory and 6 cores.</p>

<p>DiskPressure is a case where disk usage of the containers increases over a large margin such that node on which pod is running faces crunch of disk availability. This crunch would be something like <strong>&lt;5-10%</strong> of total availability.</p>
<p>In such an event, kubelet sets the DiskPressure status on the node(which inturn is not-ready for scheduling) so newer pods are not scheduled and pods are evicted(which are re-scheduled to other availability) to meet uptime of pods.</p>
<p>Most common cases of facing diskpressure is missing log rotation(debug logs) and other case would be large data being written on a node with limited disk.</p>
<p>Edit: My answer is generic and not specific to spark scenario.</p>
