title: Spark + EMR using Amazon's "maximizeResourceAllocation" setting does not use all cores/vcores
body: <p>I'm running an EMR cluster (version emr-4.2.0) for Spark using the Amazon specific <code>maximizeResourceAllocation</code> flag as documented <a href="http://docs.aws.amazon.com/ElasticMapReduce/latest/ReleaseGuide/emr-spark-configure.html" rel="noreferrer">here</a>.  According to those docs, "this option calculates the maximum compute and memory resources available for an executor on a node in the core node group and sets the corresponding spark-defaults settings with this information".</p>
  
  <p>I'm running the cluster using m3.2xlarge instances for the worker nodes.  I'm using a single m3.xlarge for the YARN master - the smallest m3 instance I can get it to run on, since it doesn't do much.</p>
  
  <p>The situation is this:  When I run a Spark job, the number of requested cores for each executor is 8. (I only got this after configuring <code>"yarn.scheduler.capacity.resource-calculator": "org.apache.hadoop.yarn.util.resource.DominantResourceCalculator"</code> which isn't actually in the documentation, but I digress).  This seems to make sense, because according to <a href="https://aws.amazon.com/ec2/instance-types/" rel="noreferrer">these docs</a> an m3.2xlarge has 8 "vCPUs".  However, on the actual instances themselves, in <code>/etc/hadoop/conf/yarn-site.xml</code>, each node is configured to have <code>yarn.nodemanager.resource.cpu-vcores</code> set to <code>16</code>.  I would (at a guess) think that must be due to hyperthreading or perhaps some other hardware fanciness.  </p>
  
  <p>So the problem is this:  when I use <code>maximizeResourceAllocation</code>, I get the number of "vCPUs" that the Amazon Instance type has, which seems to be only half of the number of configured "VCores" that YARN has running on the node; as a result, the executor is using only half of the actual compute resources on the instance.</p>
  
  <p>Is this a bug in Amazon EMR?  Are other people experiencing the same problem?  Is there some other magic undocumented configuration that I am missing?</p>