title: Flatten a Seq of Maps to Map using Type polymorphism in Scala, Spark UDF
body: <p>I have the following function that flattens a sequence of maps of string to double. How can I make type string to double generic?</p>
  
  <pre><code>val flattenSeqOfMaps = udf { values: Seq[Map[String, Double]] =&gt; values.flatten.toMap }
  flattenSeqOfMaps: org.apache.spark.sql.expressions.UserDefinedFunction = UserDefinedFunction(&lt;function1&gt;,MapType(StringType,DoubleType,false),Some(List(ArrayType(MapType(StringType,DoubleType,false),true))))
  </code></pre>
  
  <p>I need something like, </p>
  
  <pre><code>val flattenSeqOfMaps[S,D] = udf { values: Seq[Map[S, D]] =&gt; values.flatten.toMap }
  </code></pre>
  
  <p>Thanks.</p>
  
  <p>Edit 1:
  I'm using spark 2.3. I am aware of higher order functions in spark 2.4</p>
  
  <p>Edit 2: I got a bit closer. What do I need in place of <code>f _</code> in <code>val flattenSeqOfMaps = udf { f _}</code>. Please compare <code>joinMap</code> type signature and <code>flattenSeqOfMaps</code> type signature below</p>
  
  <pre><code>scala&gt; val joinMap = udf { values: Seq[Map[String, Double]] =&gt; values.flatten.toMap }
  joinMap: org.apache.spark.sql.expressions.UserDefinedFunction = UserDefinedFunction(&lt;function1&gt;,MapType(StringType,DoubleType,false),Some(List(ArrayType(MapType(StringType,DoubleType,false),true))))
  
  scala&gt; def f[S,D](values: Seq[Map[S, D]]): Map[S,D] = { values.flatten.toMap}
  f: [S, D](values: Seq[Map[S,D]])Map[S,D]
  
  scala&gt; val flattenSeqOfMaps = udf { f _}
  flattenSeqOfMaps: org.apache.spark.sql.expressions.UserDefinedFunction = UserDefinedFunction(&lt;function1&gt;,MapType(NullType,NullType,true),Some(List(ArrayType(MapType(NullType,NullType,true),true))))
  </code></pre>
  
  <p>Edit 3: the following code worked for me.</p>
  
  <pre><code>scala&gt; val flattenSeqOfMaps = udf { f[String,Double] _}
  flattenSeqOfMaps: org.apache.spark.sql.expressions.UserDefinedFunction = UserDefinedFunction(&lt;function1&gt;,MapType(StringType,DoubleType,false),Some(List(ArrayType(MapType(StringType,DoubleType,false),true))))
  </code></pre>