title: Spark lists all leaf node even in partitioned data
body: <p>I have parquet data partitioned by <code>date</code> &amp; <code>hour</code>, folder structure:</p>
  
  <pre><code>events_v3
    -- event_date=2015-01-01
      -- event_hour=2015-01-1
        -- part10000.parquet.gz
    -- event_date=2015-01-02
      -- event_hour=5
        -- part10000.parquet.gz
  </code></pre>
  
  <p>I have created a table <code>raw_events</code> via spark but when I try to query, it scans all the directories for footer and that slows down the initial query, even if I am querying only one day worth of data.</p>
  
  <p>query:
  <code>select * from raw_events where event_date='2016-01-01'</code></p>
  
  <p>similar problem : <a href="http://mail-archives.apache.org/mod_mbox/spark-user/201508.mbox/%3CCAAswR-7Qbd2tdLSsO76zyw9tvs-Njw2YVd36bRfCG3DKZrH0tw@mail.gmail.com%3E" rel="noreferrer">http://mail-archives.apache.org/mod_mbox/spark-user/201508.mbox/%3CCAAswR-7Qbd2tdLSsO76zyw9tvs-Njw2YVd36bRfCG3DKZrH0tw@mail.gmail.com%3E</a> ( but its old)</p>
  
  <p>Log:</p>
  
  <pre><code>App &gt; 16/09/15 03:14:03 main INFO HadoopFsRelation: Listing leaf files and directories in parallel under: s3a://bucket/events_v3/
  </code></pre>
  
  <p>and then it spawns 350 tasks since there are 350 days worth of data.</p>
  
  <p>I have disabled <code>schemaMerge</code>, and have also specified the schema to read as, so it can just go to the partition that I am looking at, why should it print all the leaf files ?
  Listing leaf files with 2 executors take 10 minutes, and the query actual execution takes on 20 seconds</p>
  
  <p>code sample:</p>
  
  <pre><code>val sparkSession = org.apache.spark.sql.SparkSession.builder.getOrCreate()
  val df = sparkSession.read.option("mergeSchema","false").format("parquet").load("s3a://bucket/events_v3")
      df.createOrReplaceTempView("temp_events")
      sparkSession.sql(
        """
          |select verb,count(*) from temp_events where event_date = "2016-01-01" group by verb
        """.stripMargin).show()
  </code></pre>