title: apache spark executors and data locality
body: <p>The spark literature says </p>
  
  <blockquote>
    <p>Each application gets its own executor processes, which stay up for
    the duration of the whole application and run tasks in multiple
    threads.</p>
  </blockquote>
  
  <p>And If I understand this right, In static allocation the executors are acquired by the Spark application when the Spark Context is created on all nodes in the cluster (in a cluster mode). I have a couple of questions</p>
  
  <ol>
  <li><p>If executors are acquired on all nodes and will stay allocated to
  this application during the the duration of the whole application,
  isn't there a chance a lot of nodes remain idle?</p></li>
  <li><p>What is the advantage of acquiring resources when Spark context is
      created and not in the DAGScheduler? I mean the application could be
      arbitrarily long and it is just holding the resources.</p></li>
  <li><p>So when the DAGScheduler tries to get the preferred locations and
      the executors in those nodes are running the tasks, would it
      relinquish the executors on other nodes?</p></li>
  </ol>
  
  <p>I have checked a related question
  <a href="https://stackoverflow.com/questions/43627217/does-spark-on-yarn-deal-with-data-locality-while-launching-executors">Does Spark on yarn deal with Data locality while launching executors</a></p>
  
  <p>But I'm not sure there is a conclusive answer</p>