title: Spark: How to transpose and explode columns with nested arrays
body: <p>I applied an algorithm from the question below(in NOTE) to transpose and explode nested spark dataframe.</p>
  <p>When I define <code>cols = ['a', 'b']</code> I get empty dataframe, but when I define <code>cols = ['a']</code> I get transformed dataframe for the <code>a</code> column. See section Current code below for more details. Any help would be appreciated.</p>
  <p>I'm looking for required output 2 (Transpose and Explode ) but even example of required output 1 (Transpose)  will be very useful.</p>
  <p>NOTE: This is minimum example to highlight the problem, in reality dataframe schema and arrays length vary as in the example  <a href="https://stackoverflow.com/questions/68837951/pyspark-how-to-flatten-nested-arrays-by-merging-values-in-spark">Pyspark: How to flatten nested arrays by merging values in spark</a></p>
  <p>Input df:</p>
  <pre><code>+---+------------------+--------+
  | id|                 a|       b|
  +---+------------------+--------+
  |  1|[{1, 1}, {11, 11}]|    null|
  |  2|              null|[{2, 2}]|
  +---+------------------+--------+
  
  
  root
   |-- id: long (nullable = true)
   |-- a: array (nullable = true)
   |    |-- element: struct (containsNull = true)
   |    |    |-- date: long (nullable = true)
   |    |    |-- val: long (nullable = true)
   |-- b: array (nullable = true)
   |    |-- element: struct (containsNull = true)
   |    |    |-- date: long (nullable = true)
   |    |    |-- val: long (nullable = true
  </code></pre>
  <p>Required output 1 (transpose_df):</p>
  <pre><code>+---+------+-------------------+
  | id| cols |       arrays      |
  +---+------+-------------------+
  |  1|  a   | [{1, 1}, {11, 11}]|
  |  2|  b   | [{2, 2}]          |
  +---+------+-------------------+
  </code></pre>
  <p>Required output 2 (explode_df):</p>
  <pre><code>+---+----+----+---+
  | id|cols|date|val|
  +---+----+----+---+
  |  1|   a|   1|  1|
  |  1|   a|  11| 11|
  |  2|   b|   2|  2|
  +---+----+----+---+
  </code></pre>
  <p>Current code:</p>
  <pre><code>import pyspark.sql.functions as f
  
  df = spark.read.json(sc.parallelize([
    &quot;&quot;&quot;{&quot;id&quot;:1,&quot;a&quot;:[{&quot;date&quot;:1,&quot;val&quot;:1},{&quot;date&quot;:11,&quot;val&quot;:11}]}&quot;&quot;&quot;,
    &quot;&quot;&quot;{&quot;id&quot;:2,&quot;b&quot;:[{&quot;date&quot;:2,&quot;val&quot;:2}]}}&quot;&quot;&quot;]))
  
  cols = ['a', 'b']
  
  expressions = [f.expr('TRANSFORM({col}, el -&gt; STRUCT(&quot;{col}&quot; AS cols, el.date, el.val))'.format(col=col)) for col in cols ]
  
  transpose_df = df.withColumn('arrays', f.flatten(f.array(*expressions)))
               
  explode_df = transpose_df.selectExpr('id', 'inline(arrays)')
  
  explode_df.show()
  </code></pre>
  <p>Current Outcome</p>
  <pre><code>+---+----+----+---+
  | id|cols|date|val|
  +---+----+----+---+
  +---+----+----+---+
  </code></pre>