title: How to extract data from MapType Scala Spark Column as Scala Map?
body: <p>Well, the question is pretty much that. Let me provide sample:</p>
  
  <pre><code>import org.apache.spark.sql.functions._
  import org.apache.spark.sql.{DataFrame, Column, Dataset}
  
  val data = List(
    Row("miley", 
        Map("good_songs" -&gt; "wrecking ball",
            "bad_songs" -&gt; "younger now"
           )
       ),
      Row("kesha",
          Map(
            "good_songs" -&gt; "tik tok",
            "bad_songs" -&gt; "rainbow"
          )
         )
  )
  
  val schema = List(
    StructField("singer", StringType, true),
    StructField("songs", MapType(StringType, StringType, true))
  )
  
  val someDF = spark.createDataFrame(
    spark.sparkContext.parallelize(data),
    StructType(schema)
  )
  
  
  // This returns scala.collection.Map[Nothing,Nothing]
  someDF.select($"songs").head().getMap(0) 
  
  // Therefore, this won't work:
  val myHappyMap : Map[String, String] = someDF.select($"songs").head().getMap(0)
  </code></pre>
  
  <p>I don't understand why I'm getting a <code>Map[Nothing, Nothing]</code> if I properly described my desired schema for the MapType column - not only that: when I do <code>someDF.schema</code>, what I get is
  <code>org.apache.spark.sql.types.StructType = StructType(StructField(singer,StringType,true), StructField(songs,MapType(StringType,StringType,true),true))</code>, showing that the DataFrame schema is properly set.</p>
  
  <p>I've read <a href="https://stackoverflow.com/questions/36703614/extract-or-filter-maptype-of-spark-dataframe">extract or filter MapType of Spark DataFrame
  </a>, and also <a href="https://stackoverflow.com/questions/40602606/how-to-get-keys-and-values-from-maptype-column-in-sparksql-dataframe">How to get keys and values from MapType column in SparkSQL DataFrame
  </a>. I thought the latter would solve my problem by at least being able to extract the keys and the values separately, but, still, I get the values as <code>WrappedArray(Nothing)</code>, which means it just adds extra complication for no real gain.</p>
  
  <p>What am I missing here?</p>