title: How does Spark handle Timestamp types during Pandas dataframe conversion?
body: <p>I have a pandas dataframe with timestamp columns of type pandas.tslib.Timestamp. I looked through the pyspark source code from 'createDataFrame'(<a href="https://github.com/apache/spark/blob/d492cc5a21cd67b3999b85d97f5c41c3734b1ba3/python/pyspark/sql/session.py#L515" rel="noreferrer">link to source</a>) and it seems that they convert the data to a numpy record array to a list:</p>
  
  <pre><code>data = [r.tolist() for r in data.to_records(index=False)]
  </code></pre>
  
  <p>However, timestamp types get converted in this process to a list of longs:</p>
  
  <pre><code>&gt; df = pd.DataFrame(pd.date_range(start=datetime.datetime.now(),periods=5,freq='s'))
  &gt; df
  0 2017-07-25 11:53:29.353923
  1 2017-07-25 11:53:30.353923
  2 2017-07-25 11:53:31.353923
  3 2017-07-25 11:53:32.353923
  4 2017-07-25 11:53:33.353923
  &gt; df.to_records(index=False).tolist()
  [(1500983799614193000L,), (1500983800614193000L,), (1500983801614193000L,), (1500983802614193000L,), (1500983803614193000L,)]
  </code></pre>
  
  <p>Now if I pass such a list to a RDD, do some operations(not touching the timestamp column) and then call</p>
  
  <pre><code>&gt; spark.createDataFrame(rdd,schema) // with schema mentioning that column as TimestampType
  TypeError: TimestampType can not accept object 1465197332112000000L in type &lt;type 'long'&gt;
      at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRDD.scala:193)
      at org.apache.spark.api.python.PythonRunner$$anon$1.&lt;init&gt;(PythonRDD.scala:234)
      at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:152)
      at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:63)
      at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
      at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
      at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
      at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
      at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
      at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
      at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
      at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
      at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
      at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
      at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
      at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
      at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
      at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
      at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
      at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
      at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
      at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
      at org.apache.spark.scheduler.Task.run(Task.scala:99)
      at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)
  </code></pre>
  
  <p>What should I do(before converting the list to a RDD) to preserve the datetime type. </p>
  
  <p><strong>Edit 1</strong></p>
  
  <p>A few methods I'm aware of that will involve post dataframe creation processing are:</p>
  
  <ol>
  <li><p>Add timezone information to datetime object in pandas. However this seems unnecessary and can lead to errors depending on your own timezone.</p></li>
  <li><p>Convert the long to a timestamp using the datetime library.</p></li>
  </ol>
  
  <blockquote>
    <p>Assuming tstampl is the input:  tstamp = datetime(1970, 1, 1) +
    timedelta(microseconds=tstampl/1000)</p>
  </blockquote>
  
  <ol start="3">
  <li>Convert the datetime to string on Pandas dataframe side, then cast to datetime on Spark dataframe side.</li>
  </ol>
  
  <blockquote>
    <p>As explained in Suresh's answer below</p>
  </blockquote>
  
  <p><strong>However I'm looking for a simpler way which would take care of all processing before dataframe creation itself.</strong></p>