title: Spark2 reads ORC files much slower than Spark1
body: <p>I found that Spark2 loads ORC files much slower than Spark1, and then I tried some methods to speed up Spark2, but no success. Codes show as below:</p>
  <p><em>Spark 1.5</em></p>
  
  <pre><code>    val conf = new SparkConf().setAppName(&quot;LoadOrc&quot;)
        .set(&quot;spark.serializer&quot;, &quot;org.apache.spark.serializer.KryoSerializer&quot;)
        .set(&quot;spark.akka.frameSize&quot;, &quot;512&quot;)
        .set(&quot;spark.akka.timeout&quot;,&quot;800s&quot;)
        .set(&quot;spark.storage.blockManagerHeartBeatMs&quot;, &quot;300000&quot;)
        .set(&quot;spark.kryoserializer.buffer.max&quot;,&quot;1024m&quot;)
        .set(&quot;spark.executor.extraJavaOptions&quot;, &quot;-Djava.util.Arrays.useLegacyMergeSort=true&quot;)
  
      val sc = new SparkContext(conf)
  
      val hiveContext = new HiveContext(sc)
      val start = System.nanoTime()
      val ret = hiveContext.read.orc(args(0)).count()
      val end = System.nanoTime()
      println(s&quot;count: $ret&quot;)
      println(s&quot;Time taken: ${(end - start) / 1000 / 1000} ms&quot;)
      sc.stop()
  </code></pre>
  <p><em>Spark UI</em></p>
  <p><a href="https://i.stack.imgur.com/FQQYM.png" rel="nofollow noreferrer">Spark1 UI</a></p>
  <p><em>results</em></p>
  <pre><code>count: 2290811187
  Time taken: 401063 ms
  </code></pre>
  <p><em>Spark 2</em></p>
  <pre><code>    val spark = SparkSession.builder()
        .appName(&quot;LoadOrc&quot;)
        .config(&quot;spark.serializer&quot;, &quot;org.apache.spark.serializer.KryoSerializer&quot;)
        .config(&quot;spark.akka.frameSize&quot;, &quot;512&quot;)
        .config(&quot;spark.akka.timeout&quot;,&quot;800s&quot;)
        .config(&quot;spark.storage.blockManagerHeartBeatMs&quot;, &quot;300000&quot;)
        .config(&quot;spark.kryoserializer.buffer.max&quot;,&quot;1024m&quot;)
        .config(&quot;spark.executor.extraJavaOptions&quot;, &quot;-Djava.util.Arrays.useLegacyMergeSort=true&quot;)
        .enableHiveSupport()
        .getOrCreate()
  
      println(spark.time(spark.read.format(&quot;org.apache.spark.sql.execution.datasources.orc&quot;)
        .load(args(0)).count()))
  
      spark.close()
  </code></pre>
  <p><em>Spark UI</em></p>
  <p><a href="https://i.stack.imgur.com/jlC3n.png" rel="nofollow noreferrer">Spark2 UI</a></p>
  <p><em>results</em></p>
  <pre><code>Time taken: 1384464 ms
  2290811187
  </code></pre>