title: Spark not leveraging hdfs partitioning with parquet
body: <p>I am writing a parquet file to hdfs using the following command:
  <code>df.write.mode(SaveMode.Append).partitionBy(id).parquet(path)</code></p>
  
  <p>Afterwards I am reading and filtering the file like this:</p>
  
  <pre><code>val file = sqlContext.read.parquet(folder)
  val data = file.map(r =&gt; Row(r.getInt(4).toString, r.getString(0), r.getInt(1),
      r.getLong(2), r.getString(3)))
  
  val filteredData = data.filter(x =&gt; x.thingId.equals("1"))
  filteredData.collect()
  </code></pre>
  
  <p>I would expect, that Spark would leverage the partitioning of the file and only read the partition of "thingId = 1".
  In Fact, Spark does read all partitions of the file and not only the filtered one (partition with thingId=1).
  If I look in the logs, I can see that it does read everything:</p>
  
  <blockquote>
    <p>16/03/21 01:32:33 INFO ParquetRelation: Reading Parquet file(s) from
    hdfs://sandbox.hortonworks.com/path/id=1/part-r-00000-b4e27b02-9a21-4915-89a7-189c30ca3fe3.gz.parquet
    16/03/21 01:32:33 INFO ParquetRelation: Reading Parquet file(s) from
    hdfs://sandbox.hortonworks.com/path/id=42/part-r-00000-b4e27b02-9a21-4915-89a7-189c30ca3fe3.gz.parquet
    16/03/21 01:32:33 INFO ParquetRelation: Reading Parquet file(s) from
    hdfs://sandbox.hortonworks.com/path/id=17/part-r-00000-b4e27b02-9a21-4915-89a7-189c30ca3fe3.gz.parquet
    16/03/21 01:32:33 INFO ParquetRelation: Reading Parquet file(s) from
    hdfs://sandbox.hortonworks.com/path/0833/id=33/part-r-00000-b4e27b02-9a21-4915-89a7-189c30ca3fe3.gz.parquet
    16/03/21 01:32:33 INFO ParquetRelation: Reading Parquet file(s) from
    hdfs://sandbox.hortonworks.com/path/id=26/part-r-00000-b4e27b02-9a21-4915-89a7-189c30ca3fe3.gz.parquet
    16/03/21 01:32:33 INFO ParquetRelation: Reading Parquet file(s) from
    hdfs://sandbox.hortonworks.com/path/id=12/part-r-00000-b4e27b02-9a21-4915-89a7-189c30ca3fe3.gz.parquet</p>
  </blockquote>
  
  <p>Is there anything I am missing? When I look at the docs, Spark should know based on the filter, that it should only read Partition with thingID=1. 
  Does anyone of you have an idea whats the problem?</p>