title: PySpark set local dir to avoid java.io.IOException: No space left on device
body: <p><code>Spark</code> is throwing the error <code>java.io.IOException: No space left on device</code> which I have traced to the overflow of the directory <code>/tmp</code> where <code>Spark</code> is creating temporary files.  I want to manually specify another location for these files where more space is available.  Currently using <code>PySpark 3.1.2</code> on <code>Ubuntu 20.04</code>.  Have tried the following already without success (<code>Spark</code> still writes to <code>/tmp</code>):</p>
  <pre><code>from pyspark.sql import SparkSession
  spark = SparkSession.builder.getOrCreate()
  spark.conf.set('spark.local.dirs', '/home/tmp')
  </code></pre>
  <p>have also tried instead</p>
  <pre><code>spark.conf.set('spark.local.dir', '/home/tmp')
  </code></pre>
  <p>In both cases <code>Spark</code> is ignoring the configuration change to write to another dir (<code>/home/tmp</code>) and instead is writing to the default dir as if no configuration change (<code>/tmp</code>) where there is not sufficient space.</p>