title: PySpark: How to extract variables from a struct nested in a struct inside an array?
body: <p>The following is a toy example that is a subset of my actual data's schema. I abbreviated it for brevity. </p>
  
  <p>I am looking to build a PySpark dataframe that contains 3 fields: <code>ID</code>, <code>Type</code> and <code>TIMESTAMP</code> that I would then save as a Hive Table. I am struggling with the PySpark code to extract the relevant columns.</p>
  
  <pre><code> |-- Records: array (nullable = true)
   |    |-- element: struct (containsNull = true)
   |    |    |-- FileID: long (nullable = true)
   |    |    |-- SrcFields: struct (nullable = true)
   |    |    |    |-- ID: string (nullable = true)
   |    |    |    |-- Type: string (nullable = true)
   |    |    |    |-- TIMESTAMP: string (nullable = true)
  </code></pre>
  
  <p>Thus far, I imagine my solution should look something like:</p>
  
  <pre><code>from pyspark.sql.functions import col, explode
  
  df.withColumn("values", explode("values")).select(
    "*", col("values")["name"].alias("name"), col("values")["id"].alias("id")
  )
  </code></pre>
  
  <p>However, the solution above doesn't account for the extra nesting of my use-case and I'm unable to figure out the additional syntax required. </p>