title: How can I pass extra parameters to UDFs in Spark SQL?
body: <p>I want to parse the date columns in a <code>DataFrame</code>, and for each date column, the resolution for the date may change (i.e. 2011/01/10 => 2011 /01 if the resolution is set to "Month"). </p>
  
  <p>I wrote the following code:</p>
  
  <pre><code>def convertDataFrame(dataframe: DataFrame, schema : Array[FieldDataType], resolution: Array[DateResolutionType]) : DataFrame =
  {
    import org.apache.spark.sql.functions._
    val convertDateFunc = udf{(x:String, resolution: DateResolutionType) =&gt; SparkDateTimeConverter.convertDate(x, resolution)}
    val convertDateTimeFunc = udf{(x:String, resolution: DateResolutionType) =&gt; SparkDateTimeConverter.convertDateTime(x, resolution)}
  
    val allColNames = dataframe.columns
    val allCols = allColNames.map(name =&gt; dataframe.col(name))
  
    val mappedCols =
    {
      for(i &lt;- allCols.indices) yield
      {
        schema(i) match
        {
          case FieldDataType.Date =&gt; convertDateFunc(allCols(i), resolution(i)))
          case FieldDataType.DateTime =&gt; convertDateTimeFunc(allCols(i), resolution(i))
          case _ =&gt; allCols(i)
        }
      }
    }
  
    dataframe.select(mappedCols:_*)
  
  }}
  </code></pre>
  
  <p>However it doesn't work. It seems that I can only pass <code>Column</code>s to UDFs. And I wonder if it will be very slow if I convert the <code>DataFrame</code> to <code>RDD</code> and apply the function on each row.</p>
  
  <p>Does anyone know the correct solution? Thank you! </p>