title: AM Container is running beyond virtual memory limits
body: <p>I was playing with distributed shell application (hadoop-2.0.0-cdh4.1.2).
  This is the error I'm receiving at the moment.</p>
  
  <pre><code>13/01/01 17:09:09 INFO distributedshell.Client: Got application report from ASM for, appId=5, clientToken=null, appDiagnostics=Application application_1357039792045_0005 failed 1 times due to AM Container for appattempt_1357039792045_0005_000001 exited with  exitCode: 143 due to: Container [pid=24845,containerID=container_1357039792045_0005_01_000001] is running beyond virtual memory limits. Current usage: 77.8mb of 512.0mb physical memory used; 1.1gb of 1.0gb virtual memory used. Killing container.
  Dump of the process-tree for container_1357039792045_0005_01_000001 :
  |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
  |- 24849 24845 24845 24845 (java) 165 12 1048494080 19590 /usr/java/bin/java -Xmx512m org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster --container_memory 128 --num_containers 1 --priority 0 --shell_command ping --shell_args localhost --debug
  |- 24845 23394 24845 24845 (bash) 0 0 108654592 315 /bin/bash -c /usr/java/bin/java -Xmx512m org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster --container_memory 128 --num_containers 1 --priority 0 --shell_command ping --shell_args localhost --debug 1&gt;/tmp/logs/application_1357039792045_0005/container_1357039792045_0005_01_000001/AppMaster.stdout 2&gt;/tmp/logs/application_1357039792045_0005/container_1357039792045_0005_01_000001/AppMaster.stderr 
  </code></pre>
  
  <p>The interesting part is that, there seems to be no problem with the setup, since a simple <code>ls</code> or <code>uname</code> command completed successfully and the output was available in the container2 stdout.</p>
  
  <p>Regarding the setup, <code>yarn.nodenamager.vmem-pmem-ratio</code> is <code>3</code> and the total physical memory available is 2GB, which I thinks is more than enough for example to run.</p>
  
  <p>For the command in question, the "ping localhost" generated two replies, as it can be seen from the <code>containerlogs/container_1357039792045_0005_01_000002/721917/stdout/?start=-4096.</code></p>
  
  <p>So, what could be the problem?</p>