title: Writing data on HDFS taking very long time
body: <p>Writing file on HDFS creating many part files(200) and taking very long time</p>
  
  <p>I am loading hive table(using SqlContext.sql) into Dataframe 1(6K records) and registering into temp table. I am left joining temp table with another hive table(5 Million records) and loading into dataframe 2. I am updating dataframe 1 with dataframe 2 values using left outer join and  i am trying to write dataframe_1(6K records) into HDFS file which is taking 1 hour 30 min. I have tried 
  repartition(dataframe_1.rdd.repartition(2)) 
  writing file without coalesce(1) 
  writing just one record to see if it is faster
  but did not help. Even show(10) records on dataframe_1 is taking long time.
  I am writing code in Zeppelin using spark interpreter. I do not have access to any configuration settings as well.Need to contact support team for that.</p>
  
  <p>I am new to big data platforms . Any hep would be greatly appreciated.</p>
  
  <pre><code>%spark
  
  import org.apache.hadoop.fs._
  import org.apache.hadoop.fs.FileSystem
  import org.apache.hadoop.fs.Path
  
  val fs=FileSystem.get(sc.hadoopConfiguration)
  val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)
  val Dataframe_1 = sqlContext.sql("""select * from  Hive_Table_1""")
  Dataframe_1.registerTempTable("Dataframe_table")
  
  val Dataframe_2 = sqlContext.sql("""SELECT * FROM 
  (select B.col1,    B.col2,    A.col4,    B.col3 FROM Dataframe_table A 
  left join  Hive_Table_2  B on a.col1=b.col1 and a.col2=b.col2) A """ )
  
  val Dataframe_1 = 
  Dataframe_1.join(Dataframe_2,Seq("col1","col2"),"leftouter")
  .withColumn("col3", when($"col3".isNotNull,"Path Found").otherwise("Path 
  Not Found"))
  
  Dataframe_1.coalesce(1).write.format("com.databricks.spark.csv")
  .option("header","true").save("/abc")
  </code></pre>