title: Is there an effective partitioning method when using reduceByKey in Spark?
body: <p>When I use <code>reduceByKey</code> or <code>aggregateByKey</code>, I'm confronted with partition problems. </p>
  
  <p>ex)<code>reduceBykey(_+_).map(code)</code></p>
  
  <p>Especially, if input data is skewed, the partitioning problem becomes even worse when using the above methods. </p>
  
  <p>So, as a solution to this, I use <code>repartition</code> method.</p>
  
  <p>For example, <a href="http://dev.sortable.com/spark-repartition/" rel="nofollow noreferrer">http://dev.sortable.com/spark-repartition/</a> is similar. </p>
  
  <p>This is good for partition distribution, but the<code>repartition</code> is also expensive.</p>
  
  <p>Is there a way to solve the partition problem wisely?</p>