title: spark cluster on aws emr cant find spark-env.sh
body: <p>I am playing with apache-spark on aws emr, and trying to use <a href="https://aws.amazon.com/premiumsupport/knowledge-center/emr-pyspark-python-3x/" rel="nofollow noreferrer">this</a> to set the cluster to use python3, </p>
  
  <p>I use the command as the last command in a bootstrap script</p>
  
  <pre><code>sudo sed -i -e '$a\export PYSPARK_PYTHON=/usr/bin/python3' /etc/spark/conf/spark-env.sh
  </code></pre>
  
  <p>When I use it the cluster crashes during the bootstrap with the following error. </p>
  
  <blockquote>
    <p>sed: can't read /etc/spark/conf/spark-env.sh: No such file or
    directory</p>
  </blockquote>
  
  <p>How should I set it to use python3 properly? </p>
  
  <p>This is not a duplicate <a href="https://stackoverflow.com/questions/40677713/how-to-set-pythonhashseed-on-aws-emr">of</a>, My issue is that the cluster is not finding the spark-env.sh file while bootstrapping, while the other question addresses the issue of the system not finding python3 </p>