title: Timestamp roundtrip from Spark Python to Pandas and back
body: <p>How do you do a roundtrip conversion of timestamp data from Spark Python to Pandas and back? I read data from a Hive table in Spark, want to do some calculations in Pandas, and write the results back to Hive. Only the last part is failing, converting a Pandas timestamp back to a Spark DataFrame timestamp.</p>
  <pre><code>import datetime
  import pandas as pd
  
  dates = [
      ('today', '2017-03-03 11:30:00')
    , ('tomorrow', '2017-03-04 08:00:00')
    , ('next Thursday', '2017-03-09 20:00:00')
  ]
  string_date_rdd = sc.parallelize(dates)
  timestamp_date_rdd = string_date_rdd.map(lambda t: (t[0], datetime.datetime.strptime(t[1], &quot;%Y-%m-%d %H:%M:%S&quot;)))
  timestamp_df = sqlContext.createDataFrame(timestamp_date_rdd, ['Day', 'Date'])
  timestamp_pandas_df = timestamp_df.toPandas()
  roundtrip_df = sqlContext.createDataFrame(timestamp_pandas_df)
  roundtrip_df.printSchema()
  roundtrip_df.show()
  
  root
   |-- Day: string (nullable = true)
   |-- Date: long (nullable = true)
  
  +-------------+-------------------+
  |          Day|               Date|
  +-------------+-------------------+
  |        today|1488540600000000000|
  |     tomorrow|1488614400000000000|
  |next Thursday|1489089600000000000|
  +-------------+-------------------+
  </code></pre>
  <p>At this point the roundtrip Spark DataFrame has the date column as datatype <code>long</code>. In Pyspark this can be converted back to a datetime object easily, e.g., <code>datetime.datetime.fromtimestamp(148908960000000000 / 1000000000)</code>, although the time of day is off by a few hours. How do I do this to convert the data type of the Spark DataFrame?</p>
  <p>Python 3.4.5, Spark 1.6.0</p>
  <p>Thanks,
  John</p>