title: why does not spark.executor.instances work?
body: <p>I'm using 40 r4.2xlarge slaves and one master with the same type host.
  r4.2xlarge has 8 cores with 61GB Memory.</p>
  
  <p>Given settings are:</p>
  
  <ul>
  <li>spark.executor.instances  280</li>
  <li>spark.executor.cores  1</li>
  <li>spark.executor.memory 8G  </li>
  <li>spark.driver.memory   40G</li>
  <li>spark.yarn.executor.memoryOverhead    10240</li>
  <li>spark.dynamicAllocation.enabled   false</li>
  </ul>
  
  <p>When observing a job running with this cluster in its Ganglia, overall cpu usage is around 30% only. and its resource manager "Aggregated Metrics by Executor" table shows only two executors per slave node.</p>
  
  <p>Why does this cluster run only two executors per slave node even with 280 spark.executor.instances? </p>
  
  <p>---- UPDATE ----</p>
  
  <p>I found the yarn-site.xml under /etc/hadoop/conf.empty</p>
  
  <ul>
  <li>yarn.scheduler.maximum-allocation-mb 54272</li>
  <li>yarn.scheduler.maximum-allocation-vcores 128</li>
  <li>yarn.nodemanager.resource.cpu-vcores 8</li>
  <li>yarn.nodemanager.resource.memory-mb 54272</li>
  </ul>