title: Spark groupBy vs repartition plus mapPartitions
body: <p>My dataset is ~20 millions of rows, it takes ~ 8 GB of RAM. I'm runing my job with 2 executors, 10 GB RAM per executor, 2 cores per executor. Due to further transformations, data should be cached all at once. </p>
  
  <p>I need to reduce duplicates based on 4 fields (choose any of duplicates). Two options: using <code>groupBy</code> and using <code>repartition</code> and <code>mapPartitions</code>. The second approach allows you to specify num of partitions, and could perform faster because of this in some cases, right?</p>
  
  <p><em>Could you please explain what option has better performance? Do both options has the same RAM consumption?</em></p>
  
  <p><em>Using <code>groupBy</code></em></p>
  
  <pre class="lang-java prettyprint-override"><code>dataSet
      .groupBy(col1, col2, col3, col4)
      .agg(
          last(col5),
          ...
          last(col17)
      );
  </code></pre>
  
  <p><em>Using <code>repartition</code> and <code>mapPartitions</code></em></p>
  
  <pre class="lang-java prettyprint-override"><code>dataSet.sqlContext().createDataFrame(
      dataSet
          .repartition(parallelism, seq(asList(col1, col2, col3, col4)))
          .toJavaRDD()
          .mapPartitions(DatasetOps::reduce),
      SCHEMA
  );
  
  private static Iterator&lt;Row&gt; reduce(Iterator&lt;Row&gt; itr) {
      Comparator&lt;Row&gt; comparator = (row1, row2) -&gt; Comparator
          .comparing((Row r) -&gt; r.getAs(name(col1)))
          .thenComparing((Row r) -&gt; r.getAs(name(col2)))
          .thenComparingInt((Row r) -&gt; r.getAs(name(col3)))
          .thenComparingInt((Row r) -&gt; r.getAs(name(col4)))
          .compare(row1, row2);
  
      List&lt;Row&gt; list = StreamSupport
          .stream(Spliterators.spliteratorUnknownSize(itr, Spliterator.ORDERED), false)
          .collect(collectingAndThen(toCollection(() -&gt; new TreeSet&lt;&gt;(comparator)), ArrayList::new));
  
      return list.iterator();
  }
  </code></pre>