title: spark-cmd not found when trying to run Spark in Rstudio
body: <p>I am following the instructions found <a href="http://www.r-bloggers.com/installing-and-starting-sparkr-locally-on-windows-os-and-rstudio/" rel="nofollow noreferrer">here on rbloggers</a> to set up spark on a redhat machine. I want to use Spark in RStudio.</p>
  <p>I have downloaded <code>spark-1.6.1-bin-hadoop2.6</code> and followed the instructions as and put the following line in a script in RStudio:</p>
  <pre><code># Setting SPARK_HOME
  Sys.setenv(SPARK_HOME = &quot;~/Downloads/spark-1.6.1-bin-hadoop2.6&quot;)
  
  # Setting library path
  .libPaths(c(file.path(Sys.getenv(&quot;SPARK_HOME&quot;), &quot;R&quot;, &quot;lib&quot;), .libPaths()))
  
  library(SparkR)
  
  # create a spark context
  sc &lt;- sparkR.init(master = &quot;local&quot;)
  </code></pre>
  <p>But the last line returns the following error:</p>
  <blockquote>
  <p>Launching java with spark-submit command ~/Downloads/spark-1.6.1-bin-hadoop2.6/bin/spark-submit   sparkr-shell /tmp/RtmpSwsYUW/backend_port3752546940e6</p>
  <p>sh: ~/Downloads/spark-1.6.1-bin-hadoop2.6/bin/spark-submit: No such file or directory</p>
  </blockquote>
  <p>I have tried every solution on the internet before asking this. For example:</p>
  <ul>
  <li><p>JAVA_HOME and SPARK_HOME are set.</p>
  </li>
  <li><p>Giving <code>spark-submit</code> executable by <code>chmod a+x spark-submit.cmd</code> (and also <code>chmod u+w spark-submit.cmd</code>) and did not work. (<em>of course I was in the correct library</em>)</p>
  </li>
  <li><p>Tried <code>spark-shell</code> in terminal and it works (returns a working shell ins scala)</p>
  </li>
  <li><p>Adding this before initialization:</p>
  <pre><code>  Sys.setenv(&quot;SPARK_SUBMIT_ARGS&quot;=&quot; - - master yarn-client sparkr-shell&quot;)
  </code></pre>
  </li>
  </ul>
  <p>The only issue I can think of now, is that there is no <code>sparkr-shell</code> in the directory. It is just <code>sparkr.cmd</code> and <code>sparkr2.cmd</code>. Now I am wondering is it related to spark version that I downloaded? Should I install hadoop first?</p>