title: Format TimestampType in spark DataFrame- Scala
body: <p>While I try to cast a string field to a TimestampType in Spark DataFrame, the output value is coming with microsecond precision( <code>yyyy-MM-dd HH:mm:ss.S</code>). But I need the format to be <code>yyyy-MM-dd HH:mm:ss</code> ie., excluding the microsecond precision. Also, I want to save this as a time stamp field while writing into a parquet file.
  So the datatype of my field should be a timestamp of format <code>yyyy-MM-dd HH:mm:ss</code></p>
  
  <p>I tried using TimestampType as</p>
  
  <pre><code>col("column_A").cast(TimestampType)
  or
  col("column_A").cast("timestamp")
  </code></pre>
  
  <p>to cast the field to timestamp. These are able to cast the field to timestamp but with the microsecond precision.</p>
  
  <p>Can anyone help in saving the timestamp datatype to parquet file with the required format specification.
  <br><strong>EDIT</strong>
  <br>Input:</p>
  
  <pre><code>val a = sc.parallelize(List(("a", "2017-01-01 12:02:00.0"), ("b", "2017-02-01 11:22:30"))).toDF("cola", "colb")
  scala&gt; a.withColumn("datetime", date_format(col("colb"), "yyyy-MM-dd HH:mm:ss")).show(false)
  +----+---------------------+-------------------+
  |cola|colb                 |datetime           |
  +----+---------------------+-------------------+
  |a   |2017-01-01 12:02:00.0|2017-01-01 12:02:00|
  |b   |2017-02-01 11:22:30  |2017-02-01 11:22:30|
  +----+---------------------+-------------------+
  
  
  scala&gt; a.withColumn("datetime", date_format(col("colb"), "yyyy-MM-dd HH:mm:ss")).printSchema
  root
   |-- cola: string (nullable = true)
   |-- colb: string (nullable = true)
   |-- datetime: string (nullable = true)
  </code></pre>
  
  <p>In the above, we are getting the right timestamp format, but when we print the Schema, the datetime field is of type String, but I need a timestamp type here.</p>
  
  <p>Now,if I attempt to cast the field to timestamp, the format is set to microsecond precision, which is not intended.</p>
  
  <pre><code>scala&gt; import org.apache.spark.sql.types._
  import org.apache.spark.sql.types._
  
  scala&gt; val a = sc.parallelize(List(("a", "2017-01-01 12:02:00.0"), ("b", "2017-02-01 11:22:30"))).toDF("cola", "colb")
  a: org.apache.spark.sql.DataFrame = [cola: string, colb: string]
  
  scala&gt; a.withColumn("datetime", date_format(col("colb").cast(TimestampType), "yyyy-MM-dd HH:mm:ss").cast(TimestampType)).show(false)
  +----+---------------------+---------------------+
  |cola|colb                 |datetime             |
  +----+---------------------+---------------------+
  |a   |2017-01-01 12:02:00.0|2017-01-01 12:02:00.0|
  |b   |2017-02-01 11:22:30  |2017-02-01 11:22:30.0|
  +----+---------------------+---------------------+
  
  
  scala&gt; a.withColumn("datetime", date_format(col("colb").cast(TimestampType), "yyyy-MM-dd HH:mm:ss").cast(TimestampType)).printSchema
  root
   |-- cola: string (nullable = true)
   |-- colb: string (nullable = true)
   |-- datetime: timestamp (nullable = true)
  </code></pre>
  
  <p>What I am expecting is for the format to be in <code>yyyy-MM-dd HH:mm:ss</code> and also the datatype of the field to be of <code>timestamp</code>
  Thanks in advance</p>