title: Combine 2 columns to key/value array
body: <p>My answer should be simple, but I tried many things and coudn't get the expected result. I Have 3 columns in AWS Glue.</p>
  <pre><code>identifier  brand_key   brand_value
  1           1324        Brand 1
  1           64          Brand 2
  1           262         Brand 3
  1           457822      Brand 4
  1           343         Brand 5
  1           23          Brand 6
  1           53          Brand 7
  1           6           Brand 8
  1           32          Brand 9
  1           463464      Brand 10
  1           456         Brand 11
  </code></pre>
  <p>It contains 3 columns, identifier, the brand_key and brand_value.
  THe identifier I added only to group on it. Does not have a extra function.
  Now I tried many things and this is the closest I could get.</p>
  <p>code:</p>
  <pre><code>newdf1 = newdf1.groupBy('identifier').agg( collect_set('brand_key').alias('brands') )
  </code></pre>
  <p>result:</p>
  <pre><code>[
    {
      &quot;brands&quot;: [
        &quot;Brand 1&quot;, 
        &quot;Brand 2&quot;, 
        &quot;Brand 3&quot;, 
        &quot;Brand 4&quot;, 
        &quot;Brand 5&quot;, 
        &quot;Brand 6&quot;, 
        &quot;Brand 7&quot;, 
        &quot;Brand 8&quot;, 
        &quot;Brand 9&quot;, 
        &quot;Brand 10&quot;, 
        &quot;Brand 11&quot;, 
      ]
    }
  ]
  </code></pre>
  <p>But I try to get this output:</p>
  <pre><code>[
    {
      &quot;brands&quot;: {
        &quot;1324&quot;: &quot;Brand 1&quot;, 
        &quot;64&quot;: &quot;Brand 2&quot;, 
        &quot;262&quot;: &quot;Brand 3&quot;, 
        &quot;457822&quot;: &quot;Brand 4&quot;, 
        &quot;343&quot;: &quot;Brand 5&quot;, 
        &quot;23&quot;: &quot;Brand 6&quot;, 
        &quot;53&quot;: &quot;Brand 7&quot;, 
        &quot;6&quot;: &quot;Brand 8&quot;, 
        &quot;32&quot;: &quot;Brand 9&quot;, 
        &quot;463464&quot;: &quot;Brand 10&quot;
        &quot;456&quot;: &quot;Brand 11&quot;
      }
    }
  ]
  </code></pre>
  <p>It should not be difficult right? I also tried to create a JSON field with:</p>
  <pre><code>newdf1 = newdf1.withColumn(&quot;combined&quot;,concat(lit(&quot;{\&quot;&quot;),'brand_value',lit(&quot;\&quot;:\&quot;&quot;),'brand_key',lit(&quot;\&quot;}&quot;)))
  </code></pre>
  <p>And then convert that back to columns, but also was not working like expected.</p>