title: To convert struct column names into rows in a parquet file
body: <p>I have a sample json datafile as shown below:</p>
  <pre><code>{&quot;data_id&quot;:&quot;1234&quot;,&quot;risk_characteristics&quot;:{&quot;indicators&quot;:{&quot;alcohol&quot;:true,&quot;house&quot;:true,&quot;business&quot;:true,&quot;familyname&quot;:true,&quot;swimming_pool&quot;:true}}}
  {&quot;data_id&quot;:&quot;6789&quot;,&quot;risk_characteristics&quot;:{&quot;indicators&quot;:{&quot;alcohol&quot;:true,&quot;house&quot;:true,&quot;business&quot;:false,&quot;familyname&quot;:true}}}
  {&quot;data_id&quot;:&quot;5678&quot;,&quot;risk_characteristics&quot;:{&quot;indicators&quot;:{&quot;alcohol&quot;:false}}}
  
  </code></pre>
  <p>I converted the json file to parquet and loaded into hive using below code</p>
  <pre><code>dataDF = spark.read.json(&quot;path/Datasmall.json&quot;)
  dataDF.write.parquet(&quot;data.parquet&quot;)
  parqFile = spark.read.parquet(&quot;data.parquet&quot;)
  parqFile.write.saveAsTable(&quot;indicators_table&quot;, format='parquet', mode='append', path='/externalpath/indicators_table/')
  
  from pyspark.sql import HiveContext
  hive_context = HiveContext(sc)
  fromHiveDF = hive_context.table(&quot;default.indicators_table&quot;)
  fromHiveDF.show()
  
  indicatorsDF = fromHiveDF.select('data_id', 'risk_characteristics.indicators')
  indicatorsDF.printSchema()
  
  root
   |-- data_id: string (nullable = true)
   |-- indicators: struct (nullable = true)
   |    |-- alcohol: boolean (nullable = true)
   |    |-- house: boolean (nullable = true)
   |    |-- business: boolean (nullable = true)
   |    |-- familyname: boolean (nullable = true)
  
  indicatorsDF.show()
  +-------+--------------------+
  |data_id|          indicators|
  +-------+--------------------+
  |   1234|[true, true, true...|
  |   6789|[true, false, tru...|
  |   5678|         [false,,,,]|
  +-------+--------------------+
  </code></pre>
  <p>Instead of retrieving the data as select data_id, indicators.alcohol, indicators.house etc,
  I simply want to get a parquet data file with below 3 columns only . That is - the struct fields are converted to rows under indicators_type column name.</p>
  <pre><code>data_id      indicators_type     indicators_value
  1234         alcohol             T
  1234         house               T
  1234         business            T
  1234         familyname          T
  1234         swimming_ppol       T
  6789         alcohol             T
  6789         house               F
  6789         business            T
  6789         familyname          F
  5678         alcohol             F
  </code></pre>
  <p>May I ask how to do that. I am trying to accomplish this using pyspark. Also Is there a way to achieve this without hardcoding the literal details. In my actual data the struct data can extend beyond familyname and it could be even 100 of them.</p>
  <p>Thanks a lot</p>