title: What are the best practices to partition Parquet files by timestamp in Spark?
body: <p>I'm pretty new to Spark (2 days) and I'm pondering the best way to partition parquet files.</p>
  
  <p>My rough plan ATM is:</p>
  
  <ul>
  <li>read in the source TSV files with com.databricks.spark.csv (these have a TimeStampType column)</li>
  <li>write out parquet files, partitioned by year/month/day/hour</li>
  <li>use these parquet files for all the queries that'll then be occurring in future </li>
  </ul>
  
  <p>It's been ludicrously easy (kudos to Spark devs) to get a simple version working - except for partitioning the way I'd like to. This is in python BTW: </p>
  
  <pre class="lang-python prettyprint-override"><code>input = sqlContext.read.format('com.databricks.spark.csv').load(source, schema=myschema)
  input.write.partitionBy('type').format("parquet").save(dest, mode="append")
  </code></pre>
  
  <p>Is the best approach to map the RDD, adding new columns for year, month, day, hour and then use <code>PartitionBy</code>? Then for any queries we have to manually add year/month etc?  Given how elegant I've found spark to be so far, this seems a little odd.</p>
  
  <p>Thanks</p>