title: Converting TZ timestamp string to a given format in UTC using spark and scala
body: <p>I have a column called lastModified with String as given below that represents time in GMT.
  <strong>"2019-06-24T15:36:16.000Z"</strong></p>
  
  <p>I want to format this string to the format <strong>yyyy-MM-dd HH:mm:ss</strong> in spark using scala. To achieve this, I created a dataframe with a new column <em>"ConvertedTS"</em>.
  which gives incorrect time.</p>
  
  <p>Machine from where I am running this is in America/New_York timezone.</p>
  
  <pre><code>df.withColumn("ConvertedTS", date_format(to_utc_timestamp(to_timestamp(col("lastModified"), "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'"), "America/New_York"), "yyyy-MM-dd HH:MM:SS").cast(StringType))
  </code></pre>
  
  <p>I am basically looking for formatting the result of below statement in yyyy-MM-dd HH:mm:ss</p>
  
  <pre><code>df.withColumn("LastModifiedTS", col("lastModified"))
  </code></pre>
  
  <p>One of the ways that is currently working for me is udf but as udfs are not recommended, I was looking for more of a direct expression that I can use.</p>
  
  <pre><code>val convertToTimestamp = (logTimestamp: String) =&gt; {
      println("logTimeStamp: " + logTimestamp)
      var newDate = ""
      try {
        val sourceFormat = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSSXXX")
        sourceFormat.setTimeZone(TimeZone.getTimeZone("GMT"))
        val convertedDate = sourceFormat.parse(logTimestamp)
        val destFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss")
        destFormat.setTimeZone(TimeZone.getTimeZone("GMT"))
        newDate = destFormat.format(convertedDate)
        println("newDate: " + newDate)
      } catch {
        case e: Exception =&gt; e.printStackTrace()
      }
      newDate
    }
  
    //register for sql
    EdlSparkObjects.sparkSession.sqlContext.udf.register("convertToTimestamp", convertToTimestamp)
  
    // register for scala
    def convertToTimestampUDF = udf(convertToTimestamp)
    df.withColumn("LastModifiedTS", convertToTimestampUDF(col("lastModified")))
  </code></pre>
  
  <p>Thanks for help and guidance.</p>