title: Write a pyspark.sql.dataframe.DataFrame without losing information
body: <p>I am trying to save an pyspark.sql.dataframe.DataFrame in CSV format (could also be another format, as long as it is easily readable). </p>
  
  <p>So far, I found a couple of examples to save the DataFrame. However, it is losing information everytime that I write it. </p>
  
  <p>Dataset example: </p>
  
  <pre><code># Create an example Pyspark DataFrame
  
  from pyspark.sql import Row
  
  Employee = Row("firstName", "lastName", "email", "salary")
  employee1 = Employee('A', 'AA', 'mail1', 100000)
  employee2 = Employee('B', 'BB', 'mail2', 120000 )
  employee3 = Employee('C', None, 'mail3', 140000 )
  employee4 = Employee('D', 'DD', 'mail4', 160000 )
  employee5 = Employee('E', 'EE', 'mail5', 160000 )
  
  department1 = Row(id='123', name='HR')
  department2 = Row(id='456', name='OPS')
  department3 = Row(id='789', name='FN')
  department4 = Row(id='101112', name='DEV')
  
  departmentWithEmployees1 = Row(department=department1, employees=[employee1, employee2, employee5])
  departmentWithEmployees2 = Row(department=department2, employees=[employee3, employee4])
  departmentWithEmployees3 = Row(department=department3, employees=[employee1, employee4, employee3])
  departmentWithEmployees4 = Row(department=department4, employees=[employee2, employee3])
  
  departmentsWithEmployees_Seq = [departmentWithEmployees1, departmentWithEmployees2]
  dframe = spark.createDataFrame(departmentsWithEmployees_Seq)
  </code></pre>
  
  <p>In order to save this file as CSV, I firstly tried <a href="https://stackoverflow.com/questions/31385363/how-to-export-a-table-dataframe-in-pyspark-to-csv">this solution</a>: </p>
  
  <pre><code>type(dframe)
  Out[]: pyspark.sql.dataframe.DataFrame
  dframe.write.csv('junk_mycsv.csv')
  </code></pre>
  
  <p>Unfortunately, that result in this error:   </p>
  
  <pre><code>org.apache.spark.sql.AnalysisException: CSV data source does not support struct&amp;lt;id:string,name:string&amp;gt; data type.; 
  </code></pre>
  
  <p>That is the reason why I tried another possibility, to convert the spark dataframe into a pandas dataframe, and save it then. As mentioned in <a href="https://stackoverflow.com/questions/50958721/convert-a-spark-dataframe-to-pandas-df">this</a> example. </p>
  
  <pre><code>pandas_df = dframe.toPandas()
  </code></pre>
  
  <p>Works good! However, If I show my data, it is missing data: </p>
  
  <pre><code>print(pandas_df.head())
  
  department                                          employees
  0   (123, HR)  [(A, AA, mail1, 100000), (B, BB, mail2, 120000...
  1  (456, OPS)  [(C, None, mail3, 140000), (D, DD, mail4, 1600...
  </code></pre>
  
  <p>As you can see in the snapshot below, we are missing information. Because the data should be like this:</p>
  
  <pre><code>department              employees
  0  id:123, name:HR      firstName: A, lastName: AA, email: mail1, salary: 100000
  
  # Info is missing like 'id', 'name', 'firstName', 'lastName', 'email' etc. 
  # For the complete expected example, see screenshow below. 
  </code></pre>
  
  <p><a href="https://i.stack.imgur.com/IKM5A.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/IKM5A.png" alt="Expected Data format"></a></p>
  
  <p>Just for information: I am working in Databricks, with Python. </p>
  
  <p><strong>Therefore, how can I write my data (dframe from the example above) without losing information?</strong></p>
  
  <p>Many thanks in advance!</p>
  
  <p><strong>Edit</strong>
  Adding a picture for Pault, to show the format of the csv (and the headers). </p>
  
  <p><strong>Edit2</strong> 
  Replacing the picture for example csv output: </p>
  
  <p>After running Pault's code: </p>
  
  <pre><code>from pyspark.sql.functions import to_json
  dframe.select(*[to_json(c).alias(c) for c in dframe.columns])\
      .repartition(1).write.csv("junk_mycsv.csv", header= True)
  </code></pre>
  
  <p>The output is not tidy, since most column headers are empty (due the nested format?). Only copying the first row: </p>
  
  <pre><code>department           employees              (empty ColName)     (empty ColName)   (and so on)
  {\id\":\"123\"       \"name\":\"HR\"}"     [{\firstName\":\"A\"  \"lastName\":\"AA\"    (...)
  </code></pre>