title: Spark Dataframes are getting created successfully but not able to write into the Local Disk
body: <p>I am using IntelliJ IDE for executing Spark Scala code on Microsoft Windows Platform.</p>
  
  <p>I have four Spark Dataframes of around 30000 records each and I tried to take one column from each of those Dataframes as part of my requirement.</p>
  
  <p>I used Spark SQL function to do it and it got executed successfully. When I execute DF.show() or DF.count() method, I am able to see results in the screen but when I tried to write the dataframe into my local disk (windows directory) the job is getting aborted with the below error :</p>
  
  <blockquote>
    <p>Exception in thread "main" org.apache.spark.SparkException: Job
    aborted.  at
    org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply$mcV$sp(FileFormatWriter.scala:147)
      at
    org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:121)
      at
    org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:121)
      at
    org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:57)
      at
    org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:121)
      at
    org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:101)
      at
    org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58)
      at
    org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56)
      at
    org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74)
      at
    org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)
      at
    org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)
      at
    org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:135)
      at
    org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
      at
    org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:132)
      at
    org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:113)
      at
    org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:87)
      at
    org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:87)
      at
    org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:492)
      at
    org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:215)
      at
    org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:198)
      at main.src.countFeatures2$.countFeature$1(countFeatures2.scala:118)
      at
    main.src.countFeatures2$.getFeatureAsString$1(countFeatures2.scala:32)
      at main.src.countFeatures2$.main(countFeatures2.scala:40)   at
    main.src.countFeatures2.main(countFeatures2.scala)    at
    sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)   at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)     at
    com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
    Caused by: org.apache.spark.SparkException: Job aborted due to stage
    failure: Task 0 in stage 31.0 failed 1 times, most recent failure:
    Lost task 0.0 in stage 31.0 (TID 2636, localhost, executor driver):
    java.io.IOException: (null) entry in command string: null chmod 0644
    D:\Test_Output_File2_temporary\0_temporary\attempt_20170830194047_0031_m_000000_0\part-00000-85c32c55-e12d-4433-979d-ccecb2fcd341.csv
      at
    org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:770)
      at org.apache.hadoop.util.Shell.execCommand(Shell.java:866)     at
    org.apache.hadoop.util.Shell.execCommand(Shell.java:849)  at
    org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:733)
      at
    org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:225)
      at
    org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:209)
      at
    org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)
      at
    org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:296)
      at
    org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)
      at
    org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.(ChecksumFileSystem.java:398)
      at
    org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:461)
      at
    org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440)
      at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)  at
    org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)   at
    org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)   at
    org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.getRecordWriter(TextOutputFormat.java:132)
      at
    org.apache.spark.sql.execution.datasources.csv.CsvOutputWriter.(CSVRelation.scala:208)
      at
    org.apache.spark.sql.execution.datasources.csv.CSVOutputWriterFactory.newInstance(CSVRelation.scala:178)
      at
    org.apache.spark.sql.execution.datasources.FileFormatWriter$SingleDirectoryWriteTask.(FileFormatWriter.scala:234)
      at
    org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:182)
      at
    org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:129)
      at
    org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:128)
      at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
      at org.apache.spark.scheduler.Task.run(Task.scala:99)   at
    org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
      at
    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)</p>
    
    <p>Driver stacktrace:    at
    org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1435)
      at
    org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1423)
      at
    org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1422)
      at
    scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
      at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
      at
    org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1422)
      at
    org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802)
      at
    org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802)
      at scala.Option.foreach(Option.scala:257)   at
    org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:802)
      at
    org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1650)
      at
    org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1605)
      at
    org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1594)
      at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
      at
    org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:628)
      at org.apache.spark.SparkContext.runJob(SparkContext.scala:1918)    at
    org.apache.spark.SparkContext.runJob(SparkContext.scala:1931)     at
    org.apache.spark.SparkContext.runJob(SparkContext.scala:1951)     at
    org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply$mcV$sp(FileFormatWriter.scala:127)
      ... 28 more Caused by: java.io.IOException: (null) entry in command
    string: null chmod 0644
    D:\Test_Output_File2_temporary\0_temporary\attempt_20170830194047_0031_m_000000_0\part-00000-85c32c55-e12d-4433-979d-ccecb2fcd341.csv
      at
    org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:770)
      at org.apache.hadoop.util.Shell.execCommand(Shell.java:866)     at
    org.apache.hadoop.util.Shell.execCommand(Shell.java:849)  at
    org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:733)
      at
    org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:225)
      at
    org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:209)
      at
    org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)
      at
    org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:296)
      at
    org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)
      at
    org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.(ChecksumFileSystem.java:398)
      at
    org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:461)
      at
    org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440)
      at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)  at
    org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)   at
    org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)   at
    org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.getRecordWriter(TextOutputFormat.java:132)
      at
    org.apache.spark.sql.execution.datasources.csv.CsvOutputWriter.(CSVRelation.scala:208)
      at
    org.apache.spark.sql.execution.datasources.csv.CSVOutputWriterFactory.newInstance(CSVRelation.scala:178)
      at
    org.apache.spark.sql.execution.datasources.FileFormatWriter$SingleDirectoryWriteTask.(FileFormatWriter.scala:234)
      at
    org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:182)
      at
    org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:129)
      at
    org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:128)
      at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
      at org.apache.spark.scheduler.Task.run(Task.scala:99)   at
    org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
      at
    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745) Picked up _JAVA_OPTIONS:
    -Xmx512M</p>
    
    <p>Process finished with exit code 1</p>
  </blockquote>
  
  <p>I am not able to understand where it went wrong. Can anybody explain how to overcome this issue ?</p>
  
  <p><strong>UPDATE</strong>
  Please note that I was able to write the same files until yesterday and no changes were made in my system or with the IDE's configuration. So I don't understand why it was running till yesterday and why not it is running now</p>
  
  <p>There was a similar post in this link : <a href="https://stackoverflow.com/questions/40764807/null-entry-in-command-string-exception-in-saveastextfile-on-pyspark">(null) entry in command string exception in saveAsTextFile() on Pyspark</a> but they are using pyspark on Jupiter notebook whereas my issue is with IntelliJ IDE</p>
  
  <p><strong>Super simplified code that writes the output file to local disk</strong></p>
  
  <pre><code>val Test_Output =spark.sql("select A.Col1, A.Col2, B.Col2, C.Col2, D.Col2 from A, B, C, D where A.primaryKey = B.primaryKey and B.primaryKey = C.primaryKey and C.primaryKey = D.primaryKey and D.primaryKey = A.primaryKey")
  
  val Test_Output_File = Test_Output.coalesce(1).write.format("com.databricks.spark.csv").option("header", "true").option("nullValue", "0").save("D:/Test_Output_File")
  </code></pre>