title: org.apache.spark.SparkException: Job aborted due to stage failure: Task from application
body: <p>I have a problem with running spark application on standalone cluster. (I use spark 1.1.0 version).
  I succesfully run master server by command:</p>
  
  <pre><code>bash start-master.sh 
  </code></pre>
  
  <p>Then I run one worker by command:</p>
  
  <pre><code>bash spark-class org.apache.spark.deploy.worker.Worker spark://fujitsu11:7077
  </code></pre>
  
  <p>At master’s web UI:</p>
  
  <pre><code>http://localhost:8080  
  </code></pre>
  
  <p>I see, that master and worker are running.</p>
  
  <p>Then I run my application from Eclipse Luna. I successfully connect to cluster by command</p>
  
  <pre><code>JavaSparkContext sc = new JavaSparkContext("spark://fujitsu11:7077", "myapplication");
  </code></pre>
  
  <p>And after that application works, but when program achieve following code:</p>
  
  <pre><code> JavaRDD&lt;Document&gt; collectionRdd = sc.parallelize(list);
  </code></pre>
  
  <p>It's crashing with following error message:</p>
  
  <pre><code> org.apache.spark.SparkException: Job aborted due to stage failure: Task 7 in stage 0.0 failed 4 times, most recent failure: Lost task 7.3 in stage 0.0 (TID 11, fujitsu11.inevm.ru):java.lang.ClassNotFoundException: maven.maven1.Document
   java.net.URLClassLoader$1.run(URLClassLoader.java:366)
   java.net.URLClassLoader$1.run(URLClassLoader.java:355)
   java.security.AccessController.doPrivileged(Native Method)
   java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    java.lang.ClassLoader.loadClass(ClassLoader.java:425)
      java.lang.ClassLoader.loadClass(ClassLoader.java:358)
      java.lang.Class.forName0(Native Method)
      java.lang.Class.forName(Class.java:270)
      org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:59)
      java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1612)
      java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
      java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
      java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
      java.io.ObjectInputStream.readArray(ObjectInputStream.java:1706)
      java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1344)
      java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
      java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
      java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
      java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
      java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
      java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:500)
      org.apache.spark.rdd.ParallelCollectionPartition.readObject(ParallelCollectionRDD.scala:74)
      sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      java.lang.reflect.Method.invoke(Method.java:606)
      java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
      java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
      java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
      java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
      java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
      java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
      java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
      java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
      java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
      org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
      org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
      org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159)
      java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      java.lang.Thread.run(Thread.java:744)
   Driver stacktrace:
  at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1185)
  at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1174)
  at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1173)
  at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
  at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
  at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1173)
  at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:688)
  at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:688)
  at scala.Option.foreach(Option.scala:236)
  at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:688)
  at org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1391)
  at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
  at akka.actor.ActorCell.invoke(ActorCell.scala:456)
  at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
  at akka.dispatch.Mailbox.run(Mailbox.scala:219)
  at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
  at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
  at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
  at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
  at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
  </code></pre>
  
  <p>In shell I found:</p>
  
  <pre><code>14/11/12 18:46:06 INFO ExecutorRunner: Launch command: "C:\PROGRA~1\Java\jdk1.7.0_51/bin/java"  "-cp" ";;D:\spark\bin\..\conf;D:\spark\bin\..\lib\spark-assembly-
  1.1.0-hadoop1.0.4.jar;;D:\spark\bin\..\lib\datanucleus-api-jdo-3.2.1.jar;D:\spar
  k\bin\..\lib\datanucleus-core-3.2.2.jar;D:\spark\bin\..\lib\datanucleus-rdbms-3.
  2.1.jar" "-XX:MaxPermSize=128m" "-Dspark.driver.port=50913" "-Xms512M" "-Xmx512M
  " "org.apache.spark.executor.CoarseGrainedExecutorBackend" "akka.tcp://sparkDriv
  er@fujitsu11.inevm.ru:50913/user/CoarseGrainedScheduler" "0" "fujitsu11.inevm.ru
  " "8" "akka.tcp://sparkWorker@fujitsu11.inevm.ru:50892/user/Worker" "app-2014111
  2184605-0000"
  14/11/12 18:46:40 INFO Worker: Asked to kill executor app-20141112184605-0000/0
  14/11/12 18:46:40 INFO ExecutorRunner: Runner thread for executor app-2014111218
  4605-0000/0 interrupted
  14/11/12 18:46:40 INFO ExecutorRunner: Killing process!
  14/11/12 18:46:40 INFO Worker: Executor app-20141112184605-0000/0 finished with
  state KILLED exitStatus 1
  14/11/12 18:46:40 INFO LocalActorRef: Message [akka.remote.transport.ActorTransp
  ortAdapter$DisassociateUnderlying] from Actor[akka://sparkWorker/deadLetters] to
  Actor[akka://sparkWorker/system/transports/akkaprotocolmanager.tcp0/akkaProtoco
  l-tcp%3A%2F%2FsparkWorker%40192.168.3.5%3A50955-2#1066511138] was not delivered.
  [1] dead letters encountered. This logging can be turned off or adjusted with c
  onfiguration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-
  shutdown'.
  14/11/12 18:46:40 INFO LocalActorRef: Message [akka.remote.transport.Association
  Handle$Disassociated] from Actor[akka://sparkWorker/deadLetters] to Actor[akka:/
  /sparkWorker/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2
  FsparkWorker%40192.168.3.5%3A50955-2#1066511138] was not delivered. [2] dead let
  ters encountered. This logging can be turned off or adjusted with configuration
  settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
  14/11/12 18:46:41 ERROR EndpointWriter: AssociationError [akka.tcp://sparkWorker
  @fujitsu11.inevm.ru:50892] -&gt; [akka.tcp://sparkExecutor@fujitsu11.inevm.ru:50954
  ]: Error [Association failed with [akka.tcp://sparkExecutor@fujitsu11.inevm.ru:5
  0954]] [
  akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sp
  arkExecutor@fujitsu11.inevm.ru:50954]
  Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon
  $2: Connection refused: no further information: fujitsu11.inevm.ru/192.168.3.5:5
  0954
  ]
  14/11/12 18:46:42 ERROR EndpointWriter: AssociationError [akka.tcp://sparkWorker
  @fujitsu11.inevm.ru:50892] -&gt; [akka.tcp://sparkExecutor@fujitsu11.inevm.ru:50954
  ]: Error [Association failed with [akka.tcp://sparkExecutor@fujitsu11.inevm.ru:5
  0954]] [
  akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sp
  arkExecutor@fujitsu11.inevm.ru:50954]
  Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon
  $2: Connection refused: no further information: fujitsu11.inevm.ru/192.168.3.5:5
  0954
  ]
  14/11/12 18:46:43 ERROR EndpointWriter: AssociationError [akka.tcp://sparkWorker
  @fujitsu11.inevm.ru:50892] -&gt; [akka.tcp://sparkExecutor@fujitsu11.inevm.ru:50954
  ]: Error [Association failed with [akka.tcp://sparkExecutor@fujitsu11.inevm.ru:5
  0954]] [
  akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sp
  arkExecutor@fujitsu11.inevm.ru:50954]
  Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon
  $2: Connection refused: no further information: fujitsu11.inevm.ru/192.168.3.5:5
  0954
  ]
  </code></pre>
  
  <p>In logs:</p>
  
  <pre><code>14/11/12 18:46:41 ERROR EndpointWriter: AssociationError    [akka.tcp://sparkMaster@fujitsu11:7077]     -&gt; [akka.tcp://sparkDriver@fujitsu11.inevm.ru:50913]:   Error [Association failed with [akka.tcp://sparkDriver@fujitsu11.inevm.ru:50913]] [
  akka.remote.EndpointAssociationException: Association failed with   [akka.tcp://sparkDriver@fujitsu11.inevm.ru:50913]
  Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection  refused: no further information: fujitsu11.inevm.ru/192.168.3.5:50913
  ]
  14/11/12 18:46:42 INFO Master: akka.tcp://sparkDriver@fujitsu11.inevm.ru:50913 got disassociated,   removing it.
  14/11/12 18:46:42 ERROR EndpointWriter: AssociationError [akka.tcp://sparkMaster@fujitsu11:7077] -&gt; [akka.tcp://sparkDriver@fujitsu11.inevm.ru:50913]: Error [Association failed with   [akka.tcp://sparkDriver@fujitsu11.inevm.ru:50913]] [
  akka.remote.EndpointAssociationException: Association failed with   [akka.tcp://sparkDriver@fujitsu11.inevm.ru:50913]
  Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection  refused: no further information: fujitsu11.inevm.ru/192.168.3.5:50913
  ]
  14/11/12 18:46:43 ERROR EndpointWriter: AssociationError [akka.tcp://sparkMaster@fujitsu11:7077] -&gt; [akka.tcp://sparkDriver@fujitsu11.inevm.ru:50913]: Error [Association failed with   [akka.tcp://sparkDriver@fujitsu11.inevm.ru:50913]] [
  akka.remote.EndpointAssociationException: Association failed with   [akka.tcp://sparkDriver@fujitsu11.inevm.ru:50913]
  Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection  refused: no further information: fujitsu11.inevm.ru/192.168.3.5:50913
  ]
  </code></pre>
  
  <p>I googled a lot but I have no idea whats wrong...
  I found a bit similar discussion here:</p>
  
  <p><a href="https://github.com/datastax/spark-cassandra-connector/issues/187">https://github.com/datastax/spark-cassandra-connector/issues/187</a></p>
  
  <p>But it doesn't solve my problem...</p>
  
  <p>Somebody knows whats wrong?</p>
  
  <p>Thank You.</p>