title: Spark resources not fully allocated on Amazon EMR
body: <p>I'm trying to maximize cluster usage for a simple task.</p>
  
  <p>Cluster is 1+2 x m3.xlarge, runnning Spark 1.3.1, Hadoop 2.4, Amazon AMI 3.7</p>
  
  <p>The task reads all lines of a text file and parse them as csv.</p>
  
  <p>When I spark-submit a task as a yarn-cluster mode, I get one of the following result:</p>
  
  <ul>
  <li>0 executor: job waits infinitely until I manually kill it</li>
  <li>1 executor: job under utilize resources with only 1 machine working</li>
  <li>OOM when I do not assign enough memory on the driver</li>
  </ul>
  
  <p>What I would have expected:</p>
  
  <ul>
  <li>Spark driver run on cluster master with all memory available, plus 2 executors with 9404MB each (as defined by install-spark script).</li>
  </ul>
  
  <p>Sometimes, when I get a "successful" execution with 1 executor, cloning and restarting the step ends up with 0 executor.</p>
  
  <p>I created my cluster using this command:</p>
  
  <pre><code>aws emr --region us-east-1 create-cluster --name "Spark Test"
  --ec2-attributes KeyName=mykey 
  --ami-version 3.7.0 
  --use-default-roles 
  --instance-type m3.xlarge 
  --instance-count 3 
  --log-uri s3://mybucket/logs/ 
  --bootstrap-actions Path=s3://support.elasticmapreduce/spark/install-spark,Args=["-x"] 
  --steps Name=Sample,Jar=s3://elasticmapreduce/libs/script-runner/script-runner.jar,Args=[/home/hadoop/spark/bin/spark-submit,--master,yarn,--deploy-mode,cluster,--class,my.sample.spark.Sample,s3://mybucket/test/sample_2.10-1.0.0-SNAPSHOT-shaded.jar,s3://mybucket/data/],ActionOnFailure=CONTINUE
  </code></pre>
  
  <p>With some step variations including:</p>
  
  <p>--driver-memory 8G --driver-cores 4 --num-executors 2</p>
  
  <hr>
  
  <p>install-spark script with -x produces the following spark-defaults.conf:</p>
  
  <pre><code>$ cat spark-defaults.conf
  spark.eventLog.enabled  false
  spark.executor.extraJavaOptions         -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70
  spark.driver.extraJavaOptions         -Dspark.driver.log.level=INFO
  spark.executor.instances        2
  spark.executor.cores    4
  spark.executor.memory   9404M
  spark.default.parallelism       8
  </code></pre>
  
  <p><strong>Update 1</strong></p>
  
  <p>I get the same behavior with a generic JavaWordCount example:</p>
  
  <pre><code>/home/hadoop/spark/bin/spark-submit --verbose --master yarn --deploy-mode cluster --driver-memory 8G --class org.apache.spark.examples.JavaWordCount /home/hadoop/spark/lib/spark-examples-1.3.1-hadoop2.4.0.jar s3://mybucket/data/
  </code></pre>
  
  <p>However, if I remove the '--driver-memory 8G', the task gets assigned 2 executors and finishes correctly.  </p>
  
  <p>So, what's the matter with driver-memory preventing my task to get executors?</p>
  
  <p>Should the driver be executed on the cluster's master node alongside with Yarn master container as explained <a href="http://badrit.com:8000/blog/2015/2/29/running-spark-on-yarn#.VXXXgvnBzRZ">here</a>?</p>
  
  <p>How do I give more memory to my spark job driver? (Where collects and some other useful operations arise)</p>