title: Apache Spark Task Failure
body: <p>Why do Apache Spark tasks fail? I thought, due to the DAG, that even without caching Tasks were recomputable? I am in fact caching, and I either get a <code>filenotfoundexception</code> or the following:</p>
  
  <pre><code>Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 9238.0 failed 4 times, most recent failure: Lost task 0.3 in stage 9238.0 (TID 17337, ip-XXX-XXX-XXX.compute.internal): java.io.IOException: org.apache.spark.SparkException: Failed to get broadcast_299_piece0 of broadcast_299
      org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:930)
      org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:155)
      sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      java.lang.reflect.Method.invoke(Method.java:606)
      java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
      java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
      java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
      java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
      java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
      java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
      java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
      java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
      java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
      org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
      org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
      org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:160)
      java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      java.lang.Thread.run(Thread.java:745)
  </code></pre>
  
  <p>It's very bizarre because I have run the same program on smaller instances and I don't get the <code>filenotfoundexception - no space left on this device</code>, instead I get the above error. When I say, double the instance size, it tells me there's no space left on the device after about 1 hour of working - same program, bigger memory and it runs out of space! What gives?</p>