title: How to prevent Spark Executors from getting Lost when using YARN client mode?
body: <p>I have one Spark job which runs fine locally with less data but when I schedule it on YARN to execute I keep on getting the following error and slowly all executors get removed from UI and my job fails</p>
  <pre><code>15/07/30 10:18:13 ERROR cluster.YarnScheduler: Lost executor 8 on myhost1.com: remote Rpc client disassociated
  15/07/30 10:18:13 ERROR cluster.YarnScheduler: Lost executor 6 on myhost2.com: remote Rpc client disassociated
  </code></pre>
  <p>I use the following command to schedule Spark job in yarn-client mode</p>
  <pre><code> ./spark-submit --class com.xyz.MySpark --conf &quot;spark.executor.extraJavaOptions=-XX:MaxPermSize=512M&quot; --driver-java-options -XX:MaxPermSize=512m --driver-memory 3g --master yarn-client --executor-memory 2G --executor-cores 8 --num-executors 12  /home/myuser/myspark-1.0.jar
  </code></pre>
  <p>What is the problem here? I am new to Spark.</p>