title: How to extract all elements from array of structs?
body: <p>I have a Dataframe with different columns and where one of the column is an array of structs:</p>
  
  <pre><code>+----------+---------+--------------------------------------+
  |id        |title    |                                values|
  +----------+---------+--------------------------------------+
  |        1 | aaa     |      [{name1, id1}, {name2, id2},...]|
  |        2 | bbb     |  [{name11, id11}, {name22, id22},...]|
  </code></pre>
  
  <p>my df schema for this column looks like this:</p>
  
  <pre><code>|-- values: array (nullable = true)
  |    |-- element: struct (containsNull = true)
  |    |    |-- name: string (nullable = true)
  |    |    |-- id: long (nullable = true)
  </code></pre>
  
  <p>I want to extract each value from this array column like this:</p>
  
  <pre><code>+----------+---------+--------------+
  |id        |title    |name   |   _id|
  +----------+---------+--------------+
  |        1 | aaa     | name1 |  id1 |
  |        1 | aaa     | name2 |  id2 |
  |        2 | bbb     | name11| id11 |
  |        2 | bbb     | name22| id22 |
  </code></pre>
  
  <p>I figured out how to extract the single item of the array:</p>
  
  <pre><code>df = df.withColumn("name", df["values"].getItem(0).name)\
  .withColumn("_id", df["id"].getItem(0).id)\
  </code></pre>
  
  <p>but I don't know the way how to apply it for the whole length of array. I probably should do something like:</p>
  
  <pre><code>for index in range(len(df.values)):
      df = df.withColumn("name", df["values"].getItem(index).name)\
      .withColumn("_id", df["id"].getItem(index).id)\
  </code></pre>
  
  <p>could you please help me to solve it ?
  Thanks!</p>