title: Can we use SizeEstimator.estimate for estimating size of RDD/DataFrame?
body: <p>I have a DataFrame which will be created by hiveContext by executing a Hive SQL, the queried data should be pushed to different datastores in my case.</p>
  
  <p>The DataFrame has got thousands of partitions because of the SQL that I am trying to execute.</p>
  
  <p>To push the data onto datastores I use <code>mapPartitions()</code> and obtain connections and push the data.</p>
  
  <p>The load on the data destination is very high because of the number of partitions, I can <code>coalsec()</code> the number of partitions to a required count based on the size of DataFrame.</p>
  
  <p>The amount of data generated by the SQL is not same in all my cases. In few cases, it may be few 100s of records and in few cases it may go to few millions. Hence I would need a dynamic way to decide the number of partitions to <code>coalsec()</code>.</p>
  
  <p>After googling I could see that we can use <code>SizeEstimator.estimate()</code> to estimate the size of DataFrame and then divide the count based on some calculations to get number of partitions. But looking at the implementation of <code>SizeEstimator.estimate</code> at <a href="https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/SizeEstimator.scala" rel="noreferrer">spark's repo</a> showed me that it has been implemented for a single JVM stand point of view and should be used for objects like broadcast variables etc, but not for RDDs/DataFrames which are distributed across JVMs.</p>
  
  <p>Can anyone suggest how to resolve my issue? and please let me know if my understanding is wrong.</p>