title: Dynamic CPUs per Task in Spark
body: <p>Lets say my job performs several spark actions, where the first few are not using multiple cores for a single task so I would like each instance to perform (<code>executor.cores</code>) tasks in parallel (<code>spark.task.cpus=1</code>). </p>
  
  <p>Then suppose I have another action which can be parallelized - I'm desiring a feature where I could increase <code>spark.task.cpus</code> (say to use more cores on the executor), and perform fewer tasks simultaneously on each instance.</p>
  
  <p>My workaround right now is to save data, start a new <code>sparkContext</code> with new settings, and reload the data. </p>
  
  <p>The use case: my later actions may be unavoidable skewed and I may want to apply more than one core per task to avoid bottlenecking on such large tasks, but I don't want this to impact the earlier actions which can benefit from using 1 core per task.</p>
  
  <p>From looking around my guess is that I can't do this currently, so I'm mainly wondering if there is a a significant limitation for not allowing this. Alternatively, suggestions for how I could trick spark into achieving something similar. </p>
  
  <p>Note: Currently using 1.6.2 but willing to hear other options for Spark2+</p>