title: Pyspark: Using UDF with argument(s) to create a new column
body: <p>I have a user defined function as follows which I want to use to derive new columns in my dataframe:</p>
  
  <pre><code>def to_date_formatted(date_str, format):
      if date_str == '' or date_str is None:
          return None
      try:
          dt = datetime.datetime.strptime(date_str, format)
      except:
          return None
      return dt.date()
  
  spark.udf.register("to_date_udf", to_date_formatted, DateType())
  </code></pre>
  
  <p>I can use this by running sql like <code>select to_date_udf(my_date, '%d-%b-%y') as date</code>.  Note the ability to pass a custom format as an argument to the function</p>
  
  <p>However, I'm struggling to use it using pyspark column expression syntax, rather than sql</p>
  
  <p>I want to write something like:</p>
  
  <pre><code>df.with_column("date", to_date_udf('my_date', %d-%b-%y')
  </code></pre>
  
  <p>But this results in an error.  How can I do this?</p>
  
  <p>[Edit:  In this specific example, in Spark 2.2+ you can provide an optional format argument with the built in <code>to_date</code> function.  I'm on Spark 2.0 at the moment, so this is not possible for me.  Also worth noting I provided this as an example, but I'm interested in the general syntax for providing arguments to UDFs, rather than the specifics of date conversion]</p>