title: Spark. Simple "No space available in any of the local directories."
body: <p>Here is a simple test program. This is obviously a tiny test data program.</p>
  
  <pre><code>from pyspark.sql.types import Row
  from pyspark.sql.types import *
  import pyspark.sql.functions as spark_functions
  
  schema = StructType([
      StructField("cola", StringType()),
      StructField("colb", IntegerType()),
  ])
  
  rows = [
      Row("alpha", 1),
      Row("beta", 2),
      Row("gamma", 3),
      Row("delta", 4)
  ]
  
  data_frame = spark.createDataFrame(rows, schema)
  
  print("count={}".format(data_frame.count()))
  
  data_frame.write.save("s3a://my-bucket/test_data.parquet", mode="overwrite")
  
  print("done")
  </code></pre>
  
  <p>This fails with:</p>
  
  <pre><code>Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: No space available in any of the local directories.
      at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:366)
      at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createTmpFileForWrite(LocalDirAllocator.java:416)
  </code></pre>
  
  <p>This is running on Amazon EMR with S3 storage. There is plenty of disk space. Can anyone explain?</p>