title: Spark SQL UDF returning scala immutable Map with df.WithColumn()
body: <p>I have case class</p>
  
  <pre><code>case class MyCaseClass(City : String, Extras : Map[String, String])
  </code></pre>
  
  <p>and user defined function which returns scala.collection.immutable.Map</p>
  
  <pre><code>def extrasUdf = spark.udf.register(
     "extras_udf", 
     (age : Int, name : String) =&gt; Map("age" -&gt; age.toString, "name" -&gt; name)
  )
  </code></pre>
  
  <p>but this breaks with Exception:</p>
  
  <pre><code>import spark.implicits._
  
  spark.read.options(...).load(...)
        .select('City, 'Age, 'Name)
        .withColumn("Extras", extrasUdf('Age, 'Name))
        .drop('Age)
        .drop('Name)
        .as[MyCaseClass]
  </code></pre>
  
  <p>I should use spark sql's MapType(DataTypes.StringType, DataTypes.IntegerType)
  but I can't find any working example...</p>
  
  <p>And this works if I use scala.collection.Map but I need immutable Map</p>