title: Optimizing the number of tasks per core in Azure Databricks
body: <p>I'm using <strong>Azure Databricks</strong> to deploy some R code parallelized through several workers using <strong>SparkR</strong> and <code>gapplyCollect()</code>.</p>
  
  <hr>
  
  <h3>Project overview</h3>
  
  <ul>
  <li>I have <strong>10000+ similar data sources</strong> generating a lot of transactional information to be analyzed on a daily basis;</li>
  <li>I have an <strong>R function</strong> that analyzes all the information of <strong>1 data source at a time</strong>, giving me some valuable insights about that specific data source;</li>
  <li>So, every day I need to <strong>execute my R function 10000+ times</strong> for analyzing all my data.</li>
  </ul>
  
  <h3>Code Logic</h3>
  
  <ol>
  <li>Read all the data (from a relational DB) as a <code>SparkDataframe</code></li>
  <li><code>groupBy()</code> the <code>SparkDataframe</code> based on my data source column <em>(data is evenly distributed by the data source column)</em></li>
  <li>Use <code>gapplyCollect()</code> on the <code>GroupedData</code> result of last step, for applying my R function on each data partition.
  
  <ul>
  <li>The result of each execution is a small R <code>DataFrame</code> with a few rows (<em>dim == (5,5)</em>).</li>
  <li>All of the results are joined by <code>gapplyCollect()</code> execution, generating a small R <code>DataFrame</code> <em>(&lt;100k numerical rows)</em> that consolidates all the results.</li>
  </ul></li>
  <li>Persist the result on my <strong>DBFS</strong>.</li>
  </ol>
  
  <hr>
  
  <h1>My issue</h1>
  
  <p>In my dev environment I'm making some experimentation parallelizing the analysis of 250 data sources, and I'm using 4 small workers for that <em>(VM type: Standard_F4s)</em>.</p>
  
  <p><code>gapplyCollect()</code> is sending my R function to the workers, but... Is it possible to customize the maximum number of active tasks to be executed per worker? As a default, I'm seeing that <strong>Databricks</strong> is allowing 5 Active Tasks per worker.</p>
  
  <p><a href="https://i.stack.imgur.com/ho1jk.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/ho1jk.png" alt="Azure Databricks maximum tasks per worker"></a></p>
  
  <ul>
  <li>For example: How can I allow 8 tasks to be executed in parallel on each worker? Is <a href="https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-submit.html" rel="nofollow noreferrer"><code>spark-submit</code></a> suitable for this task?</li>
  </ul>
  
  <p>I've never used <code>spark-submit</code> and I didn't find a good documentation for using it on <strong>Azure Databricks</strong>.</p>
  
  <ul>
  <li><p>I'll bring this code to production using a daily scheduled job. In the job, can I use for example the <code>spark-submit</code> feature to change the <code>--executor-cores</code> option?</p></li>
  <li><p>If yes, how do I guarantee that I'm installing the <code>forecast</code> CRAN Library in all my job driver + workers, since <strong>Azure Databricks</strong> don't allow me to define libraries in the GUI when using the <code>spark-submit</code>?</p></li>
  </ul>