title: I am getting the executor running beyond memory limits when running big join in spark
body: <p>I am getting the following error in the driver of a big join on spark.</p>
  
  <p>We have 3 nodes with 32GB of ram and total input size of join is 150GB. (The same app is running properly when input file size is 50GB)</p>
  
  <p>I have set storage.memoryFraction to 0.2 and shuffle.memoryFraction to 0.2. But still keep on getting the running beyong physical limits error.</p>
  
  <blockquote>
    <p>15/04/07 19:58:17 INFO yarn.YarnAllocator: Container marked as failed:
    container_1426882329798_0674_01_000002. Exit status: 143. Diagnostics:
    Container
    [pid=51382,containerID=container_1426882329798_0674_01_000002] is
    running beyond physical memory limits. Current usage: 16.1 GB of 16 GB
    physical memory used; 16.8 GB of 33.6 GB virtual memory used. Killing
    container. Dump of the process-tree for
    container_1426882329798_0674_01_000002 :
            |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES)
    FULL_CMD_LINE
            |- 51387 51382 51382 51382 (java) 717795 50780 17970946048 4221191 /usr/jdk64/jdk1.7.0_45/bin/java -server
    -XX:OnOutOfMemoryError=kill %p -Xms14336m -Xmx14336m -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+StartAttachListener -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9010 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dlog4j.configuration=file:/softwares/log4j.properties -Djava.io.tmpdir=/hadoop/yarn/local/usercache/hdfs/appcache/application_1426882329798_0674/container_1426882329798_0674_01_000002/tmp
    -Dspark.driver.port=20763 -Dspark.ui.port=0 -Dspark.yarn.app.container.log.dir=/hadoop/yarn/log/application_1426882329798_0674/container_1426882329798_0674_01_000002
    org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url
    akka.tcp://sparkDriver@maxiq2.augmentiq.in:20763/user/CoarseGrainedScheduler
    --executor-id 1 --hostname maxiq1.augmentiq.in --cores 4 --app-id application_1426882329798_0674 --user-class-path
    file:/hadoop/yarn/local/usercache/hdfs/appcache/application_1426882329798_0674/container_1426882329798_0674_01_000002/<strong>app</strong>.jar</p>
  </blockquote>
  
  <p>Please help me out with this?</p>