<p>I am running Spark on Kubernetes cluster. While re-partitioning data with many partitions, forcing it to have only 1 file per partition, I am getting my pods evicted.</p>
<p>The error is as following:</p>
<pre><code>The node was low on resource: ephemeral-storage. Container sosreport-spark-cluster-opendatahub-w was using 56291400Ki, which exceeds its request of 0.

</code></pre>
<p>My Spark configs are:</p>
<pre><code>def create_spark_config(spark_cluster, executor_memory='16g', executor_cores='4', max_cores='16'):
    print('Spark cluster is: {}'.format(spark_cluster))
    sc_conf = (
        pyspark.SparkConf().setMaster(spark_cluster) \
        .set('spark.driver.host', HOSTNAME) \
        .set('spark.driver.port', 42000) \
        .set('spark.driver.bindAddress', '0.0.0.0') \
        .set('spark.driver.blockManager.port', 42100) \
        .set('spark.executor.memory', '1536M') \
        .set('spark.executor.cores', '2') \
        .set('spark.sql.parquet.enableVectorizedReader', True) \
        .set('spark.kubernetes.memoryOverheadFactor', '0.20')
    )
    return sc_conf
</code></pre>
<p>This is how I am re-partitioning the data:</p>
<pre><code>def save_repartitioned_dataframe(bucket_name, df):
    dest_path = form_path_string(bucket_name, repartitioned_data=True)
    print('Trying to save repartitioned data at: {}'.format(dest_path))
    df.repartition(1, &quot;created_year&quot;, &quot;created_month&quot;, &quot;created_day&quot;).write.partitionBy(
        &quot;created_year&quot;, &quot;created_month&quot;, &quot;created_day&quot;).mode(&quot;overwrite&quot;).parquet(dest_path)
    print('Data repartitioning complete with at the following location: ')
    print(dest_path)
    _, count, distinct_count, num_partitions = read_dataframe_from_bucket(bucket_name, repartitioned_data=True)
    return count, distinct_count, num_partitions
</code></pre>

Aviral Srivastava
<p>Your problem likely isn't the fact that you're using ephemeral storage but the fact that you're send your entire data to a single worker.</p>
<p>&quot;.repartition(1, &quot;created_year&quot;, &quot;created_month&quot;, &quot;created_day&quot;)&quot;</p>
<p>You're combining all of your data into a single spark partition which then writes to all partitions.</p>
<p>What you probably want todo is do a global sort by the partition key and then do a write.partionBy.  What this does is send most of the data to single table partition to a limited number of spark partitions. (typically a single spark partition if your partitions are small)</p>
<p>this typically looks like...</p>
<pre><code>    df.orderBy(&quot;partitionCol&quot;)
      .write
      .partitionBy(&quot;partitionCol&quot;)
      .insertInto(&quot;my_table&quot;)
</code></pre>
