title: Specify format of Timestamp written by pyspark
body: <p>Background is a simply pyspark programme that I developed on 1.6 using databricks csv read/writer, and all was happy. My dataframe had a timestamp column, which was written out in a standard <code>YYYY-MM-DD HH24:MI:SS</code> format. </p>
  
  <pre><code>foo,bar,2016-10-14 14:30:31.985 
  </code></pre>
  
  <p>Now I’m running it on EMR with Spark 2, and the timestamp column is being written as an <strong>epoch in microseconds</strong>. This causes a problem because the target (Redshift) can’t natively handle this (only seconds or milliseconds).</p>
  
  <pre><code>foo,bar,1476455559456000
  </code></pre>
  
  <p>Looking at <a href="http://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.DataFrameWriter.csv" rel="nofollow">the docs</a>, it seems I should be able to specify the format used with <code>timestampFormat</code>, but I just get an error : </p>
  
  <pre><code>TypeError: csv() got an unexpected keyword argument 'timestampFormat'
  </code></pre>
  
  <p>Am I calling this wrong, or does the option not exist? Any other way to cleanly get my timestamp data out in a format that's <em>not</em> microseconds (milli would be fine, or any other standard time format really)</p>
  
  <hr>
  
  <p>Simple code to reproduce: </p>
  
  <pre><code>df = sqlContext.createDataFrame([('foo','bar')]).withColumn('foo',pyspark.sql.functions.current_timestamp())
  df.printSchema()
  df.show()
  
  # Use the new Spark 2 native method
  df.write.csv(path='/tmp/foo',mode='overwrite')
  
  # Use the databricks CSV method, pre Spark 2
  df.write.save(path='/tmp/foo2',format='com.databricks.spark.csv',mode='overwrite')
  </code></pre>