title: Spark: No effect of cores per executors on application runtime
body: <p>I am testing the effect different number of cores per executors (<code>--executor-cores</code>) has on the run-time for SVD on Spark. With the <code>--executor-cores</code> fixed the number of partitions of the main data RDD is varied. However, there does not seem to be a significant change in SVD compute times for different <code>--executor-cores</code> for a given number of RDD partitions. This is a bit confusing.</p>
  
  <p>My environment is:</p>
  
  <ul>
  <li>Spark Cluster with 3 nodes (32 cores and 32GB memory per node). Each node runs 1 Worker.</li>
  <li>spark.max.cores = 96</li>
  <li>Cluster manager= <code>Standalone</code></li>
  <li>deploy mode = <code>client</code></li>
  </ul>
  
  <p>I have plotted the results for <code>--executor-cores = [4, 16]</code> and as one can see, for a given partition size there is not much difference between the compute times when the partition size increases. So my questions are: </p>
  
  <ul>
  <li>What is the effect of setting the number of cores per executor? </li>
  <li>Cores per executor does have a significant effect on runtime but only for small partition sizes and not for large ones, why?</li>
  <li>Does it affect parallelism in any way (I am not sure it does)? </li>
  </ul>
  
  <p><a href="https://i.stack.imgur.com/3OvnX.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/3OvnX.png" alt="enter image description here"></a></p>