title: Apache Spark: setting executor instances
body: <p>I run my Spark application on YARN with parameters:</p>
  
  <p>in spark-defaults.conf:</p>
  
  <pre><code>spark.master yarn-client
  spark.driver.cores 1
  spark.driver.memory 1g
  spark.executor.instances 6
  spark.executor.memory 1g
  </code></pre>
  
  <p>in yarn-site.xml:</p>
  
  <pre><code>yarn.nodemanager.resource.memory-mb 10240
  </code></pre>
  
  <p>All other parameters are set to default.</p>
  
  <p>I have a 6-node cluster and the Spark Client component is installed on each node. 
  Every time I run the application there are only 2 executors and 1 driver visible in the Spark UI. Executors appears on different nodes.</p>
  
  <p>Why can't Spark create more executors? Why are only 2 instead of 6?</p>
  
  <p>I found a very similar question: <a href="https://stackoverflow.com/questions/29940711/apache-spark-setting-executor-instances-does-not-change-the-executors">Apache Spark: setting executor instances does not change the executors</a>, but increasing the memoty-mb parameter didn't help in my case.</p>