title: Spark: How to split struct type into multiple columns?
body: <p>I know this question has been asked many times on Stack Overflow and has been satisfactorily answered in most posts, but I'm not sure if this is the best way in my case.
  I have a Dataset that has several struct types embedded in it:</p>
  
  <pre><code>root
   |-- STRUCT1: struct (nullable = true)
   |    |-- FIELD_1: string (nullable = true)
   |    |-- FIELD_2: long (nullable = true)
   |    |-- FIELD_3: integer (nullable = true)
   |-- STRUCT2: struct (nullable = true)
   |    |-- FIELD_4: string (nullable = true)
   |    |-- FIELD_5: long (nullable = true)
   |    |-- FIELD_6: integer (nullable = true)
   |-- STRUCT3: struct (nullable = true)
   |    |-- FIELD_7: string (nullable = true)
   |    |-- FIELD_8: long (nullable = true)
   |    |-- FIELD_9: integer (nullable = true)
   |-- ARRAYSTRUCT4: array (nullable = true)
   |    |-- element: struct (containsNull = true)
   |    |    |-- FIELD_10: integer (nullable = true)
   |    |    |-- FIELD_11: integer (nullable = true)
  
  +-------+------------+------------+------------------+
  |STRUCT1| STRUCT2    | STRUCT3    | ARRAYSTRUCT4     |
  +-------+------------+------------+------------------+
  |[1,2,3]|[aa, xx, yy]|[p1, q2, r3]|[[1a, 2b],[3c,4d]]|
  +-------+------------+------------+------------------+
  </code></pre>
  
  <p>I want to convert this into:</p>
  
  <p><P>
  1. A dataset where the structs are expanded into columns.<br>
  2. A data set where the array (ARRAYSTRUCT4) is exploded into rows.
  </p></p>
  
  <pre><code>root
   |-- FIELD_1: string (nullable = true)
   |-- FIELD_2: long (nullable = true)
   |-- FIELD_3: integer (nullable = true)
   |-- FIELD_4: string (nullable = true)
   |-- FIELD_5: long (nullable = true)
   |-- FIELD_6: integer (nullable = true)
   |-- FIELD_7: string (nullable = true)
   |-- FIELD_8: long (nullable = true)
   |-- FIELD_9: integer (nullable = true)
   |-- FIELD_10: integer (nullable = true)
   |-- FIELD_11: integer (nullable = true)
  
  +-------+------------+------------+---------+     ---------+----------+
  |FIELD_1| FIELD_2    | FIELD_3    | FIELD_4 |     |FIELD_10| FIELD_11 |
  +-------+------------+------------+---------+ ... ---------+----------+
  |1      |2           |3           |  aa     |     |  1a    |  2b      |
  +-------+------------+------------+-----------------------------------+
  </code></pre>
  
  <p>To achieve this, I could use:</p>
  
  <pre><code>val expanded = df.select("STRUCT1.*", "STRUCT2.*", "STRUCT3.*", "STRUCT4")
  </code></pre>
  
  <p>followed by an explode:</p>
  
  <pre><code>val exploded = expanded.select(explode(expanded("STRUCT4")))
  </code></pre>
  
  <p>However, I was wondering if there's a more functional way to do this, especially the select. I could use <code>withColumn</code> as below:</p>
  
  <pre><code>data.withColumn("FIELD_1", $"STRUCT1".getItem(0))
        .withColumn("FIELD_2", $"STRUCT1".getItem(1))
        .....
  </code></pre>
  
  <p>But I have 80+ columns. Is there a better way to achieve this?</p>