title: Scala Spark: Flatten Array of Key/Value structs
body: <p>I have an input dataframe which contains an array-typed column. Each entry in the array is a struct consisting of a key (one of about four values) and a value. I want to turn this into a dataframe with one column for each possible key, and nulls where that value is not in the array for that row. Keys are never duplicated in any of the arrays, but they may be out of order or missing.</p>
  
  <p>So far the best I've got is</p>
  
  <pre><code>val wantedCols =df.columns
    .filter(_ != arrayCol)
    .filter(_ != "col")
  val flattened = df
          .select((wantedCols.map(col(_)) ++ Seq(explode(col(arrayCol)))):_*)
          .groupBy(wantedCols.map(col(_)):_*)
          .pivot("col.key")
          .agg(first("col.value"))
  </code></pre>
  
  <p>This does exactly what I want, but it's hideous and I have no idea what the ramifactions of grouping on every-column-but-one would be. What's the RIGHT way to do this?</p>
  
  <p>EDIT: Example input/output:</p>
  
  <pre><code>case class testStruct(name : String, number : String)
  val dfExampleInput = Seq(
  (0, "KY", Seq(testStruct("A", "45"))),
  (1, "OR", Seq(testStruct("A", "30"), testStruct("B", "10"))))
  .toDF("index", "state", "entries")
  .show
  
  +-----+-----+------------------+
  |index|state|           entries|
  +-----+-----+------------------+
  |    0|   KY|         [[A, 45]]|
  |    1|   OR|[[A, 30], [B, 10]]|
  +-----+-----+------------------+
  
  val dfExampleOutput = Seq(
    (0, "KY", "45", null),
    (1, "OR", "30", "10"))
    .toDF("index", "state", "A", "B")
    .show
  
  +-----+-----+---+----+
  |index|state|  A|   B|
  +-----+-----+---+----+
  |    0|   KY| 45|null|
  |    1|   OR| 30|  10|
  +-----+-----+---+----+
  </code></pre>
  
  <p>FURTHER EDIT:</p>
  
  <p>I submitted a solution myself (see below) that handles this well so long as you know the keys in advance (in my case I do.) If finding the keys is an issue, another answer holds code to handle that.</p>