title: Are failed spark executors a cause for concern?
body: <p>I understand that Apache Spark is designed around resilient data structures, but are failures expected during a running system or does this typically indicate a problem?</p>
  
  <p>As I begin to scale the system out to different configurations, I see <code>ExecutorLostFailure</code> and <code>No more replicas</code> (See below). The system recovers and the program finishes. </p>
  
  <p>Should I be concerned with this, and are there typically things we can do to avoid this; or is this expected as the number of executors grow?</p>
  
  <pre><code>18/05/18 23:59:00 WARN TaskSetManager: Lost task 87.0 in stage 4044.0 (TID 391338, ip-10-0-0-68.eu-west-1.compute.internal, executor 11): ExecutorLostFailure (executor 11 exited caused by one of the running tasks) Reason: Container marked as failed: container_1526667532988_0010_01_000012 on host: ip-10-0-0-68.eu-west-1.compute.internal. Exit status: -100. Diagnostics: Container released on a *lost* node
  18/05/18 23:59:00 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_193_7 !
  18/05/18 23:59:00 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_582_50 !
  18/05/18 23:59:00 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_401_91 !
  18/05/18 23:59:00 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_582_186 !
  18/05/18 23:59:00 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_115_139 !
  </code></pre>