title: Possible causes of performance difference between two very similar Spark Dataframes
body: <p>I am working on improving the performance of some Spark operations for a recommendation engine I am prototyping. I have stumbled upon a significant performance differences between to DataFrames I am using. Below the results of describe() on both.</p>
  
  <p>df1 (fast, numPartitions = 4):</p>
  
  <pre><code>+-------+------------------+--------------------+
  |summary|           item_id|          popularity|
  +-------+------------------+--------------------+
  |  count|            187824|              187824|
  |   mean| 96693.34836868558|                 1.0|
  | stddev|55558.023793621316|5.281958866780519...|
  |    min|                 0|  0.9999999999999998|
  |    max|            192806|                 1.0|
  +-------+------------------+--------------------+
  </code></pre>
  
  <p>df2 (approx. 10x slower, numPartitions = ±170):</p>
  
  <pre><code>+-------+-----------------+-----------------+
  |summary|          item_id|            count|
  +-------+-----------------+-----------------+
  |  count|           187824|           187824|
  |   mean|96693.34836868558|28.70869537439305|
  | stddev|55558.02379362146|21.21976457710462|
  |    min|                0|                1|
  |    max|           192806|              482|
  +-------+-----------------+-----------------+
  </code></pre>
  
  <p>Both DataFrames are cached, the same size in terms of rows (187824) and columns (2), and have an identical <code>item_id</code> column. The main difference is that frame 1 contains float in the second column, whereas frame 2 contains integer.</p>
  
  <p>It seems as if every operation for DataFrame 2 is much slower, ranging from simply the <code>.describe().show()</code> operation, to more elaborate <code>.subtract().subtract().take()</code>. In the latter case DataFrame 2 takes 18 seconds, as opposed to 2 seconds for frame 1 (almost 10 times slower!).</p>
  
  <p>I don't know where to begin to look for an explanation of the cause of this difference. Any tips or nudges in the right direction are greatly appreciated.</p>
  
  <p><strong>UPDATE</strong>: as proposed by Viacheslav Rodionov the number of partitions of the dataframes seems to be the cause of the performance issues with df2.</p>
  
  <p>Digging deeper, both dataframes are a result of <code>.groupBy().agg().sortBy()</code> operations on the same original dataframe. The <code>.groupBy().agg()</code> operation yields 200 partitions, and then <code>.sortBy()</code> returns respectively 4 and ±170 partitions, why could this be?</p>