title: Python: How to convert Pyspark column to date type if there are null values
body: <p>In pyspark, I have a dataframe that has dates that get imported as strings.  There are null values in these dates-as-strings columns.  I'm trying to convert these columns into date type columns, but I keep getting errors.  Here's a small example of the dataframe:</p>
  
  <pre><code>+--------+----------+----------+
  |DeviceId|   Created| EventDate|
  +--------+----------+----------+
  |       1|      null|2017-03-09|
  |       1|      null|2017-03-09|
  |       1|2017-03-09|2017-03-09|
  |       1|2017-03-15|2017-03-15|
  |       1|      null|2017-05-06|
  |       1|2017-05-06|2017-05-06|
  |       1|      null|      null|
  +--------+----------+----------+
  </code></pre>
  
  <p>When there are no null values, I have found that this code below will work to convert the data types:</p>
  
  <pre><code>dt_func =  udf (lambda x: datetime.strptime(x, '%Y-%m-%d'), DateType())    
  df = df.withColumn('Created', dt_func(col('Created')))
  </code></pre>
  
  <p>Once I add null values it crashes.  I've tried to modify the udf to account for nulls as follows:</p>
  
  <pre><code>import numpy as np
  def convertDatetime(x):
      return sf.when(x.isNull(), 'null').otherwise(datetime.strptime(x, '%Y-%m-%d'))
  dt_func =  udf(convertDatetime, DateType())
  </code></pre>
  
  <p>I also tried filling the nulls with an arbitrary date-string, converting the columns to dates, and then trying to replace the arbitrary fill date with nulls as below:</p>
  
  <pre><code>def dt_conv(df, cols, form = '%Y-%m-%d', temp_plug = '1900-01-01'):
      df = df.na.fill(temp_plug)
      dt_func =  udf (lambda x: datetime.strptime(x, form), DateType())
  
      for col_ in cols:
          df = df.withColumn(col_, dt_func(col(col_)))
      df = df.replace(datetime.strptime(temp_plug, form), 'null')
      return df
  </code></pre>
  
  <p>However, this method gives me this error</p>
  
  <pre><code>ValueError: to_replace should be a float, int, long, string, list, tuple, or dict
  </code></pre>
  
  <p>Can someone help me figure this out?</p>