title: Submitting Spark job to Amazon EMR
body: <p>I'm about to try EMR and henceforth going trough the documentation right now. I'm rather a bit confused by the submit process. </p>
  
  <p><strong>1) Where are the spark Libraries</strong> </p>
  
  <p>From the Spark documentation we find:</p>
  
  <p><strong>- spark.yarn.jars:</strong> 
  List of libraries containing Spark code to distribute to YARN containers. By default, Spark on YARN will use Spark jars installed locally, but the Spark jars can also be in a world-readable location on HDFS. This allows YARN to cache it on nodes so that it doesn't need to be distributed each time an application runs. To point to jars on HDFS, for example, set this configuration to hdfs:///some/path. Globs are allowed.</p>
  
  <ul>
  <li>(a) I wonder how is that set with EMR i.e. is it set up by EMR or do I have to set that up myself ?</li>
  </ul>
  
  <p><strong>2) How to the --master parameter works ?</strong></p>
  
  <p>From the spark documentation we have:</p>
  
  <p><strong>- --master</strong>:Unlike other cluster managers supported by Spark in which the master’s address is specified in the --master parameter, in YARN mode the ResourceManager’s address is picked up from the Hadoop configuration. Thus, the --master parameter is yarn.</p>
  
  <ul>
  <li>(a) Is that set up by EMR directly ?</li>
  </ul>
  
  <p><strong>3) Is there a way to submit the application from by terminal, or is the only way to actually deploy the jar on S3 ?</strong> Can I log on to the master and do the submit from there ? Will all the Env variable necessary for the submit-script to work ready (see previous question) ? What is the most productive way to do this submit ?</p>