title: Spark SQL queries on Partitioned Data
body: <p>I have setup a Spark 1.3.1 application that collects event data.  One of the attributes is a timestamp called 'occurredAt'.   Im intending to partition the event data into parquet files on a filestore and in accordance with the documentation (<a href="https://spark.apache.org/docs/1.3.1/sql-programming-guide.html#partition-discovery" rel="nofollow">https://spark.apache.org/docs/1.3.1/sql-programming-guide.html#partition-discovery</a>) it indicates that time based values are not supported only string and int,  so i've split the date into Year, Month, Day values and partitioned as follows</p>
  
  <pre><code>events
    |---occurredAtYear=2015
    |   |---occurredAtMonth=07
    |   |   |---occurredAtDay=16
    |   |   |   |---&lt;parquet-files&gt;
    ...
  </code></pre>
  
  <p>I then load the parquet file from the root path /events</p>
  
  <pre><code>sqlContext.parquetFile('/var/tmp/events')
  </code></pre>
  
  <p>Documentation says:</p>
  
  <blockquote>
    <p>'Spark SQL will automatically extract the partitioning information
    from the paths'</p>
  </blockquote>
  
  <p>However my query </p>
  
  <pre><code>SELECT * FROM events where occurredAtYear=2015
  </code></pre>
  
  <p>Fails miserably saying spark cannot resolve 'occurredAtYear'</p>
  
  <p>I can see the schema for all other aspects of the event and can do queries on those attributes, but printSchema does not list occurredAtYear/Month/Day on the schema at all?  What am I missing to get partitioning working appropriately.</p>
  
  <p>Cheers</p>