title: How to union in a way that will group "date" into different columns instead of combining everything in the same column
body: <p>I'm very new to pyspark. Here is the little situation, I created a dataframe for each file (9 in total, each file represent the counts for each month), then I need to union them all in one big df. The thing is I need it to come out like this with each month is its own separate column.</p>
  <pre><code>name_id |  2020_01 | 2020_02 | 2020_03
     1         23       43534     3455
     2         12       34534    34534
     3         2352     32525       23
  </code></pre>
  <p>However, with my current code, it put all months under the same column. I've been searching on the internet for a long time, but could not find anything to solve this (maybe I need groupby, but not sure how to do that). Below is my code. Thanks!</p>
  <pre><code>df1=spark.read.format(&quot;parquet&quot;).load(&quot;dbfs:&quot;)
  df2=spark.read.format(&quot;parquet&quot;).load(&quot;dbfs:&quot;)
  df3=spark.read.format(&quot;parquet&quot;).load(&quot;dbfs:&quot;)
  df4=spark.read.format(&quot;parquet&quot;).load(&quot;dbfs:&quot;)
  df5=spark.read.format(&quot;parquet&quot;).load(&quot;dbfs:&quot;)
  df6=spark.read.format(&quot;parquet&quot;).load(&quot;dbfs:&quot;)
  df7=spark.read.format(&quot;parquet&quot;).load(&quot;dbfs:&quot;)
  df8=spark.read.format(&quot;parquet&quot;).load(&quot;dbfs:&quot;)
  df9=spark.read.format(&quot;parquet&quot;).load(&quot;dbfs:&quot;)
  
  #union all 9 files
  union_all=df1.unionAll(df2).unionAll(df3).unionAll(df4).unionAll(df5).unionAll(df6).unionAll(df7).unionAll(df8).unionAll(df9)
  </code></pre>
  <p>Here is the current output</p>
  <pre><code>name_id | count | date
  1          23     2020_01
  2          12     2020_01
  1         43534   2020_02
  2         34534   2020_02
  </code></pre>