title: how to change header of a data frame with another data frame header?
body: <p>I have a data set which looks like this </p>
  
  <pre><code>LineItem.organizationId|^|LineItem.lineItemId|^|StatementTypeCode|^|LineItemName|^|LocalLanguageLabel|^|FinancialConceptLocal|^|FinancialConceptGlobal|^|IsDimensional|^|InstrumentId|^|LineItemSequence|^|PhysicalMeasureId|^|FinancialConceptCodeGlobalSecondary|^|IsRangeAllowed|^|IsSegmentedByOrigin|^|SegmentGroupDescription|^|SegmentChildDescription|^|SegmentChildLocalLanguageLabel|^|LocalLanguageLabel.languageId|^|LineItemName.languageId|^|SegmentChildDescription.languageId|^|SegmentChildLocalLanguageLabel.languageId|^|SegmentGroupDescription.languageId|^|SegmentMultipleFundbDescription|^|SegmentMultipleFundbDescription.languageId|^|IsCredit|^|FinancialConceptLocalId|^|FinancialConceptGlobalId|^|FinancialConceptCodeGlobalSecondaryId|^|FFAction|!|
  Japan|^|1507101869432|^|4295876606|^|1|^|BAL|^|Cash And Deposits|^|null|^|null|^|ACAE|^|false|^|null|^|null|^|null|^|null|^|false|^|null|^|null|^|null|^|null|^|505126|^|505074|^|null|^|null|^|null|^|null|^|null|^|null|^|null|^|3018759|^|null|^|I|!|
  </code></pre>
  
  <p>And this is how i load data with auto discover schema </p>
  
  <pre><code>val df1With_ = df.toDF(df.columns.map(_.replace(".", "_")): _*)
  val column_to_keep = df1With_.columns.filter(v =&gt; (!v.contains("^") &amp;&amp; !v.contains("!") &amp;&amp; !v.contains("_c"))).toSeq
  val df1result = df1With_.select(column_to_keep.head, column_to_keep.tail: _*)
  </code></pre>
  
  <p>Now i have another data frame on which i do join operation and finally i create a data frame which writes output to csv file .</p>
  
  <p>Final data frame looks like this </p>
  
  <pre><code>val dfMainOutputFinal = dfMainOutput.select($"DataPartition", $"StatementTypeCode",concat_ws("|^|", dfMainOutput.schema.fieldNames.filter(_ != "DataPartition").map(c =&gt; col(c)): _*).as("concatenated"))
  
  val dfMainOutputFinalWithoutNull = dfMainOutputFinal.withColumn("concatenated", regexp_replace(col("concatenated"), "null", ""))
  
  dfMainOutputFinalWithoutNull.write.partitionBy("DataPartition","StatementTypeCode")
    .format("csv")
    .option("nullValue", "")
    .option("header","true")
    .option("codec", "gzip")
    .save("output")
  </code></pre>
  
  <p>Now in my output file i see my header as only  <code>concatenated</code> which is expected .</p>
  
  <p>Now my question is is there anyway to change header of my final output as header of <code>df1result</code> data frame </p>