title: Convert csv.gz files into Parquet using Spark
body: <p>I need to implement converting csv.gz files in a folder, both in AWS S3 and HDFS, to Parquet files using Spark (Scala preferred). One of the columns of the data is a timestamp and I only have a week of dataset. The timestamp format is:</p>
  
  <p>'yyyy-MM-dd hh:mm:ss'</p>
  
  <p>The output that I desire is that for every day, there is a folder (or partition) where the Parquet files for that specific date is located. So there would 7 output folders or partitions.</p>
  
  <p>I only have a faint idea of how to do this, only sc.textFile is on my mind. Is there a function in Spark that can convert to Parquet? How do I implement this in S3 and HDFS? </p>
  
  <p>Thanks for you help.</p>