<p>Questions related to cores vs executors are asked number of times on SO.</p>

<p><a href="https://stackoverflow.com/questions/24622108/apache-spark-the-number-of-cores-vs-the-number-of-executors">Apache Spark: The number of cores vs. the number of executors</a></p>

<p>As each case is different, I'm asking similar question again.</p>

<p>I'm running a cpu intensive application with same number of cores with different executors. Below are the observations.</p>

<p><strong>Resource Manager :</strong> Kubernetes</p>

<p><strong>Case 1:</strong> Executors - 6, Number of cores for each executor -2, Executor Memory - 3g, Amount of Data processing ~ 10GB, Partitions -36, Job Duration : <strong>75 mins</strong></p>

<p><strong>Case 2:</strong> Executors - 4, Number of cores for each executor -3, Executor Memory - 3g, Amount of Data processing ~ 10GB, Partitions -36, Job Duration : <strong>101 mins</strong></p>

<p>As per the above link anything less than 5 cores per executor is good for IO operations.</p>

<p>In both my cases cores are same(12), however both jobs took different times. Any thoughts?</p>

<p><strong>Updated</strong></p>

<p><strong>Case 3:</strong> Executors - 12, Number of cores for each executor -1, Executor Memory - 3g, Amount of Data processing ~ 10GB, Partitions -36, Job Duration : <strong>81 mins</strong></p>

data_addict
<p>There are many possible solutions, 
First of all not all nodes are born equal, it might be one of the jobs got unlucky and got a slow node.
Seconds if you perform shuffle operations having more nodes but the same computation power will really slow your job. after all in a shuffle operation all of your information will be eventually stored on a single node. having this node with less data before hand and less power will slow the operation.
But I suspect that even without shuffle operations more nodes will be a bit slower, as there is a higher chance of a single node having more work to do from other nodes.</p>

<p>Explanation:</p>

<p>Lets say I have a single node and 10 hour of work and 10 cores, so I know it will take 1 hour
but if I have 2 nodes, with 5 cores each and lets say the dataset was partitioned in a way one node has 5.5 hours of work  and the other 4.5 hours , the job length will be 1.1 hours.</p>

<p>There is always overhead price to pay for distributed computing. so Its usually faster to do work with the same resources on a single machine.</p>

<p>Hope what I tried to say is clear.</p>
