title: Why RDD calculating count take so much time
body: <p>(English is not my first language so please excuse any mistakes)</p>
  
  <p>I use SparkSQL reading 4.7TB data from hive table, and performing a count operation. It takes about 1.6 hours to do that. While reading directly from HDFS txt file and performing count, it takes only 10 minutes. The two jobs used same resources and parallelism. Why RDD count takes so much time?</p>
  
  <p>The hive table has about 3000 thousand columns, and maybe serialization is costly. I checked the spark UI and each tasks read about 240MB data and take about 3.6 minutes to execute. I can't believe that serialization overhead is so expensive.</p>
  
  <p>Reading from hive(taking 1.6 hours):</p>
  
  <pre class="lang-scala prettyprint-override"><code>val sql = s"SELECT * FROM xxxtable"
  val hiveData = sqlContext.sql(sql).rdd
  val count = hiveData.count()
  </code></pre>
  
  <p>Reading from hdfs(taking 10 minutes):</p>
  
  <pre class="lang-scala prettyprint-override"><code>val inputPath = s"/path/to/above/hivetable"
  val hdfsData = sc.textFile(inputPath)
  val count = hdfsData.count()
  </code></pre>
  
  <p>While using SQL count, it still takes 5 minutes:</p>
  
  <pre class="lang-scala prettyprint-override"><code>val sql = s"SELECT COUNT(*) FROM xxxtable"
  val hiveData = sqlContext.sql(sql).rdd
  hiveData.foreach(println(_))
  </code></pre>