title: Understanding Spark container failure
body: <p>I have a <strong>Spark</strong> job that runs on AWS EMR. It often fails after a few steps and gives error messages like:</p>
  
  <pre><code>2016-08-18 23:29:50,167 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl (Container Monitor): Memory usage of ProcessTree 7031 for container-id container_: 48.6 GB of 52 GB physical memory used; 52.6 GB of 260 GB virtual memory used  
  2016-08-18 23:29:53,191 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl (Container Monitor): Memory usage of ProcessTree 7031 for container-id container_: 1.2 MB of 52 GB physical memory used; 110.4 MB of 260 GB virtual memory used   
  2016-08-18 23:29:56,208 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl (Container Monitor): Memory usage of ProcessTree 7031 for container-id container_: 1.2 MB of 52 GB physical memory used; 110.4 MB of 260 GB virtual memory used    
  2016-08-18 23:29:56,385 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor (ContainersLauncher #0): Exit code from container container_ is : 52   
  2016-08-18 23:29:56,387 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor (ContainersLauncher #0): Exception from container-launch with container ID: container_ and exit code: 52   
  org.apache.hadoop.util.Shell$ExitCodeException: 
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
    at org.apache.hadoop.util.Shell.run(Shell.java:418)
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
    at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:200)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
  2016-08-18 23:29:56,393 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor (ContainersLauncher #0): 
  2016-08-18 23:29:56,455 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch (ContainersLauncher #0): Container exited with a non-zero exit code 52
  </code></pre>
  
  <p>From what I can find it seems this is an OOM error. Looking earlier in the logs I can see this:</p>
  
  <pre><code>2016-08-18 23:19:00,462 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl (Container Monitor): Memory usage of ProcessTree 7031 for container-id container_: 53.6 GB of 52 GB physical memory used; 104.4 GB of 260 GB virtual memory used   
  2016-08-18 23:19:03,480 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl (Container Monitor): Memory usage of ProcessTree 7031 for container-id container_: 53.9 GB of 52 GB physical memory used; 104.4 GB of 260 GB virtual memory used    
  2016-08-18 23:19:06,498 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl (Container Monitor): Memory usage of ProcessTree 7031 for container-id container_: 53.9 GB of 52 GB physical memory used; 104.4 GB of 260 GB virtual memory used
  2016-08-18 23:19:09,516 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl (Container Monitor): Memory usage of ProcessTree 7031 for container-id container_: 53.8 GB of 52 GB physical memory used; 104.4 GB of 260 GB virtual memory used
  </code></pre>
  
  <p>My questions:</p>
  
  <ol>
  <li>Is this an OOM?</li>
  <li>Why the 10 minute gap between the over-allocation (?) and the exit?</li>
  <li>Do I need more executors in my job or do I have an incorrect param somewhere?</li>
  </ol>