title: Calculate time between two dates in pyspark
body: <p>Hoping this is fairly elementary. I have a Spark dataframe containing a Date column, I want to add a new column with number of days since that date. Google fu is failing me.</p>
  
  <p>Here's what I've tried:</p>
  
  <pre><code>from pyspark.sql.types import *
  import datetime
  today = datetime.date.today()
  
  schema = StructType([StructField("foo", DateType(), True)])
  l = [(datetime.date(2016,12,1),)]
  df = sqlContext.createDataFrame(l, schema)
  df = df.withColumn('daysBetween',today - df.foo)
  df.show()
  </code></pre>
  
  <p>it fails with error:</p>
  
  <blockquote>
    <p>u"cannot resolve '(17212 - foo)' due to data type mismatch: '(17212 -
    foo)' requires (numeric or calendarinterval) type, not date;"</p>
  </blockquote>
  
  <p>I've tried fiddling around but gotten nowhere. I can't think that this is too hard. Can anyone help?</p>