title: How to set SPARK_LOCAL_DIRS parameter using spark-env.sh file
body: <p>I am trying to change the location spark writes temporary files to. Everything I've found online says to set this by setting the <code>SPARK_LOCAL_DIRS</code> parameter in the <code>spark-env.sh</code> file, but I am not having any luck with the changes actually taking effect. </p>
  
  <p>Here is what I've done:</p>
  
  <ol>
  <li>Created a 2-worker test cluster using Amazon EC2 instances. I'm using spark 2.2.0 and the R <code>sparklyr</code> package as a front end. The worker nodes are spun up using an auto scaling group.</li>
  <li>Created a directory to store temporary files in at <code>/tmp/jaytest</code>. There is one of these in each worker and one in the master.</li>
  <li>Puttied into the spark master machine and the two workers, navigated to <code>home/ubuntu/spark-2.2.0-bin-hadoop2.7/conf/spark-env.sh</code>, and modified the file to contain this line: <code>SPARK_LOCAL_DIRS="/tmp/jaytest"</code></li>
  </ol>
  
  <p>Permissions for each of the <code>spark-env.sh</code> files are <code>-rwxr-xr-x</code>, and for the jaytest folders are <code>drwxrwxr-x</code>.</p>
  
  <p>As far as I can tell this is in line with all the advice I've read online. However, when I load some data into the cluster it still ends up in <code>/tmp</code>, rather than <code>/tmp/jaytest</code>.</p>
  
  <p>I have also tried setting the <code>spark.local.dir</code> parameter to the same directory, but also no luck.</p>
  
  <p>Can someone please advise on what I might be missing here?</p>
  
  <p>Edit: I'm running this as a standalone cluster (as the answer below indicates that the correct parameter to set depends on the cluster type). </p>