title: How to use Spark SQL DataFrame with flatMap?
body: <p>I am using the Spark Scala API. I have a Spark SQL DataFrame (read from an Avro file) with the following schema:</p>
  
  <pre><code>root
  |-- ids: array (nullable = true)
  |    |-- element: map (containsNull = true)
  |    |    |-- key: integer
  |    |    |-- value: string (valueContainsNull = true)
  |-- match: array (nullable = true)
  |    |-- element: integer (containsNull = true)
  </code></pre>
  
  <p>Essentially 2 columns [ ids: List[Map[Int, String]], match: List[Int] ]. Sample data that looks like:</p>
  
  <pre><code>[List(Map(1 -&gt; a), Map(2 -&gt; b), Map(3 -&gt; c), Map(4 -&gt; d)),List(0, 0, 1, 0)]
  [List(Map(5 -&gt; c), Map(6 -&gt; a), Map(7 -&gt; e), Map(8 -&gt; d)),List(1, 0, 1, 0)]
  ...
  </code></pre>
  
  <p>What I would like to do is <code>flatMap()</code> each row to produce 3 columns [<strong>id</strong>, <strong>property</strong>, <strong>match</strong>]. Using the above 2 rows as the input data we would get:</p>
  
  <pre><code>[1,a,0]
  [2,b,0]
  [3,c,1]
  [4,d,0]
  [5,c,1]
  [6,a,0]
  [7,e,1]
  [8,d,0]
  ...
  </code></pre>
  
  <p>and then <code>groupBy</code> the <code>String</code> <strong>property</strong> (ex: a, b, ...) to produce <code>count("property")</code> and <code>sum("match")</code>:</p>
  
  <pre><code> a    2    0
   b    1    0
   c    2    2
   d    2    0
   e    1    1
  </code></pre>
  
  <p>I would want to do something like:</p>
  
  <pre><code>val result = myDataFrame.select("ids","match").flatMap( 
      (row: Row) =&gt; row.getList[Map[Int,String]](1).toArray() )
  result.groupBy("property").agg(Map(
      "property" -&gt; "count",
      "match" -&gt; "sum" ) )
  </code></pre>
  
  <p>The problem is that the <code>flatMap</code> converts DataFrame to RDD. Is there a good way to do a <code>flatMap</code> type operation followed by <code>groupBy</code> using DataFrames?</p>