title: Expand array-of-structs into columns in PySpark
body: <p>I have a Spark dataframe, originating from Google Analytics, that looks like the following:</p>
  
  <pre><code>id     customDimensions (Array&lt;Struct&gt;)
  100    [ {"index": 1, "value": "Earth"}, {"index": 2, "value": "Europe"}]
  101    [ {"index": 1, "value": "Mars" }]
  </code></pre>
  
  <p>I also have a "custom dimensions metadata" dataframe that looks like this:</p>
  
  <pre><code>index   name
  1       planet
  2       continent
  </code></pre>
  
  <p>I'd to use the indexes in the metadata df in order to expand my custom dimensions into columns. The result should look like the following:</p>
  
  <pre><code>id     planet     continent
  100    Earth      Europe
  101    Mars       null
  </code></pre>
  
  <hr>
  
  <p>I have tried the following approach, and it works fine, however it is extremely non-performant. I'd like to know if there's a better approach.</p>
  
  <pre><code># Select the two relevant columns
  cd = df.select('id', 'customDimensions')
  
  # Explode customDimensions so that each row now has a {index, value}
  cd = cd.withColumn('customDimensions', F.explode(cd.customDimensions))
  
  # Put the index and value into their own columns
  cd = cd.select('id', 'customDimensions.index', 'customDimensions.value')
  
  # Join with metadata to obtain the name from the index
  metadata = metadata.select('index', 'name')
  cd = (cd
           .join(metadata, cd.index == metadata.index, 'left')
           .drop(metadata.index))
  
  # Pivot cd so that each row has the id, and we have columns for each custom dimension
  piv = cd.groupBy('id').pivot('name').agg(F.first(F.col('value')))
  
  # Join back to restore the other columns
  return df.join(piv, df.id == piv.id).drop(piv.id)
  </code></pre>
  
  <hr>
  
  <p>Assumptions:</p>
  
  <ul>
  <li>There are up to 250 custom dimension indexes, and the names are only known through the metadata dataframe</li>
  <li>The original dataframe has several other columns that I would like to maintain (hence the join at the end of my solution)</li>
  </ul>