title: SparkSQL - Difference between two time stamps in minutes
body: <p>I'm trying to convert the difference in minutes between two timestamps in the form <code>MM/dd/yyyy hh:mm:ss AM/PM</code>. I'm new to working with SparkSQL and tried using the basic <code>datediff</code> function that other SQL syntaxes support I.e <code>datediff(minute,start_time,end_time)</code>, but that yielded the error: </p>
  
  <pre><code>org.apache.spark.sql.AnalysisException: cannot resolve '`minute`' given input columns: [taxisub.tpep_dropoff_datetime, taxisub.DOLocationID, taxisub.improvement_surcharge, taxisub.VendorID, taxisub.trip_distance, taxisub.tip_amount, taxisub.tolls_amount, taxisub.payment_type, taxisub.fare_amount, taxisub.tpep_pickup_datetime, taxisub.total_amount, taxisub.store_and_fwd_flag, taxisub.extra, taxisub.passenger_count, taxisub.PULocationID, taxisub.mta_tax, taxisub.RatecodeID]; line 1 pos 153;
  </code></pre>
  
  <p>It seems that the <code>minute</code> argument is not supported by datediff for sparkSQL. The query I currently have is:</p>
  
  <pre><code>spark.sqlContext.sql("Select to_timestamp(tpep_pickup_datetime,'MM/dd/yyyy hh:mm:ss') as pickup,to_timestamp(tpep_dropoff_datetime,'MM/dd/yyyy hh:mm:ss') as dropoff, datediff(to_timestamp(tpep_pickup_datetime,'MM/dd/yyyy hh:mm:ss'),to_timestamp(tpep_dropoff_datetime,'MM/dd/yyyy hh:mm:ss')) as diff from taxisub ").show()
  </code></pre>
  
  <p>And my results are:</p>
  
  <pre><code>+-------------------+-------------------+----+
  |             pickup|            dropoff|diff|
  +-------------------+-------------------+----+
  |2018-12-15 08:53:20|2018-12-15 08:57:57|   0|
  |2018-12-15 08:03:08|2018-12-15 08:07:30|   0|
  |2018-12-15 08:28:34|2018-12-15 08:33:31|   0|
  |2018-12-15 08:37:53|2018-12-15 08:43:47|   0|
  |2018-12-15 08:51:02|2018-12-15 08:55:54|   0|
  |2018-12-15 08:03:47|2018-12-15 08:03:50|   0|
  |2018-12-15 08:45:21|2018-12-15 08:57:08|   0|
  |2018-12-15 08:04:47|2018-12-15 08:29:05|   0|
  |2018-12-15 08:01:22|2018-12-15 08:12:15|   0|
  +-------------------+-------------------+----+
  </code></pre>
  
  <p>I'm assuming the default for <code>datediff</code> is difference in Days, given the 0 in the results. Is there an additional argument/function that I should be using to determine the difference in minutes between these two timestamps?</p>
  
  <p>Thanks in advance.</p>