title: How to save array data frame output from spark xml in csv format
body: <p>I have deleted two of my question because i thought i was too big and i could not explained it neatly .</p>
  
  <p>So i am trying to make it simple this time .</p>
  
  <p>So i have an complex nested xml .
  I am parsing it in spark scala and i have to save all the data from the xml into text file .</p>
  
  <p><strong>NOTE:I need to save the data into text files because later i have to join this data with another file which is in text format .
  Also can i join my csv file format with json or perquet file format ?If yes then i may not need to convert my xml into text file .</strong></p>
  
  <p>This is my code where i am trying to save the xml into csv file but as csv does not allow to save array type so i am getting error .</p>
  
  <p>I am looking for some solution where i would be able to extarct all elements of an array and save it into text file .</p>
  
  <pre><code>def main(args: Array[String]) {
  
      val conf = new SparkConf().setAppName("XML").setMaster("local");
      val sc = new SparkContext(conf); //Creating spark context
      val sqlContext = new org.apache.spark.sql.SQLContext(sc)
  
      val df = sqlContext.read.format("com.databricks.spark.xml").option("rowTag", "env:Body").load("C://Users//u6034690//Desktop//SPARK//trfsmallfffile//XML")
      val resDf = df.withColumn("FlatType", explode(df("env:ContentItem"))).select("FlatType.*")
  
      resDf.repartition(1).write
        .format("csv")//This does not support for array Type
        .option("timestampFormat", "yyyy/MM/dd HH:mm:ss ZZ")
        .option("nullValue", "")
        .option("delimiter", "\t")
        .option("quote", "\u0000")
        .option("header", "true")
        .save("C://Users//u6034690//Desktop//SPARK//trfsmallfffile//XML//output")
  
      // val resDf = df.withColumn("FlatType", when(df("env:ContentItem").isNotNull, explode(df("env:ContentItem"))))
    }
  </code></pre>
  
  <p>This is producing me below output before saving </p>
  
  <pre><code>+---------+--------------------+
  |  _action|            env:Data|
  +---------+--------------------+
  |   Insert|[fun:FundamentalD...|
  |Overwrite|[sr:FinancialSour...|
  |Overwrite|[pe:FinancialPeri...|
  |Overwrite|[pe:FinancialPeri...|
  |Overwrite|[pe:FinancialPeri...|
  |Overwrite|[pe:FinancialPeri...|
  |Overwrite|[pe:FinancialPeri...|
  |Overwrite|[pe:FinancialPeri...|
  |Overwrite|[pe:FinancialPeri...|
  |Overwrite|[pe:FinancialPeri...|
  |Overwrite|[pe:FinancialPeri...|
  |Overwrite|[pe:FinancialPeri...|
  |Overwrite|[fl:FinancialLine...|
  |Overwrite|[fl:FinancialLine...|
  |Overwrite|[fl:FinancialLine...|
  |Overwrite|[fl:FinancialLine...|
  |Overwrite|[fl:FinancialLine...|
  |Overwrite|[fl:FinancialLine...|
  |Overwrite|[fl:FinancialLine...|
  |Overwrite|[fl:FinancialLine...|
  +---------+--------------------+
  </code></pre>
  
  <p>Foe each unique <code>env:Data</code> i am expecting unique file that can be done using partition but how can i save it in text file .</p>
  
  <p>I have to save all the elements from the array i mean all columns .</p>
  
  <p>I hope this time i am making my question clear .</p>
  
  <p>If required i can update schema also .</p>