title: Yarn - why doesn't task go out of heap space but container gets killed?
body: <p>If a YARN container grows beyond its heap size setting, the map or reduce task will fail, with an error similar to the one below:</p>
  
  <pre><code>2015-02-06 11:58:15,461 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Container [pid=10305,containerID=container_1423215865404_0002_01_000007] is running beyond physical memory limits. 
  Current usage: 42.1 GB of 42 GB physical memory used; 42.9 GB of 168 GB virtual memory used. Killing container.
  Dump of the process-tree for container_1423215865404_0002_01_000007 :
          |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
          |- 10310 10305 10305 10305 (java) 1265097 48324 46100516864 11028122 /usr/java/default/bin/java -server -XX:OnOutOfMemoryError=kill %p -Xms40960m -Xmx40960m -XX:MaxPermSize=128m -Dspark.sql.shuffle.partitions=20 -Djava.io.tmpdir=/data/yarn/datanode/nm-local-dir/usercache/admin/appcache/application_1423215865404_0002/container_1423215865404_0002_01_000007/tmp org.apache.spark.executor.CoarseGrainedExecutorBackend akka.tcp://sparkDriver@marx-61:56138/user/CoarseGrainedScheduler 6 marx-62 5
          |- 10305 28687 10305 10305 (bash) 0 0 9428992 318 /bin/bash -c /usr/java/default/bin/java -server -XX:OnOutOfMemoryError='kill %p' -Xms40960m -Xmx40960m  -XX:MaxPermSize=128m -Dspark.sql.shuffle.partitions=20 -Djava.io.tmpdir=/data/yarn/datanode/nm-local-dir/usercache/admin/appcache/application_1423215865404_0002/container_1423215865404_0002_01_000007/tmp org.apache.spark.executor.CoarseGrainedExecutorBackend akka.tcp://sparkDriver@marx-61:56138/user/CoarseGrainedScheduler 6 marx-62 5 1&gt; /opt/hadoop/logs/userlogs/application_1423215865404_0002/container_1423215865404_0002_01_000007/stdout 2&gt; /opt/hadoop/logs/userlogs/application_1423215865404_0002/container_1423215865404_0002_01_000007/stderr
  </code></pre>
  
  <p>It is interesting to note that all stages complete, just when save as sequence file is called, it fails. The executor is not using up the heap space, wonder what else is eating it up?</p>