title: YARN nodemanager error while running basic sparkpi example
body: <p>I am running a basic spark program to test my YARN setup with Spark. I am running a job similar to the example one on the website.</p>
  
  <pre><code> spark-submit --master yarn --deploy-mode cluster --num-executors 75 -- 
   executor-cores 2 --executor-memory 6g --class 
  org.apache.spark.examples.JavaSparkPi  
   /home/spark/examples/jars/spark_examples.jar 1000
  </code></pre>
  
  <p>However, the job never terminates and the nodemanagers on the different nodes show this error: </p>
  
  <p><code>2020-03-16 14:27:42,917 WARN  org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: couldn't find container container_1584386586744_0001_01_000319 while processing FINISH_CONTAINERS event</code></p>
  
  <p>I am not sure whats causing this. Any advice is appreciated.</p>
  
  <p>Here is the yarn-site.xml file for the standalone cluster(causes error):</p>
  
  <pre><code>&lt;configuration&gt;
  &lt;property&gt;
   &lt;name&gt;yarn.nodemanager.aux-services&lt;/name&gt;
   &lt;value&gt;mapreduce_shuffle&lt;/value&gt;
   &lt;/property&gt;
   &lt;property&gt;
   &lt;name&gt;yarn.nodemanager.auxservices.mapreduce.shuffle.class&lt;/name&gt;
   &lt;value&gt;org.apache.hadoop.mapred.ShuffleHandler&lt;/value&gt;
   &lt;/property&gt;
   &lt;property&gt;
          &lt;name&gt;yarn.acl.enable&lt;/name&gt;
          &lt;value&gt;0&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
          &lt;name&gt;yarn.resourcemanager.hostname&lt;/name&gt;
          &lt;value&gt;172.16.1.1&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
          &lt;name&gt;yarn.resourcemanager.webapp.address&lt;/name&gt;
          &lt;value&gt;10.66.4.100:8088&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
          &lt;name&gt;yarn.nodemanager.aux-services&lt;/name&gt;
          &lt;value&gt;mapreduce_shuffle&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
          &lt;name&gt;yarn.resourcemanager.resource-tracker.address&lt;/name&gt;
          &lt;value&gt;172.16.1.1&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
          &lt;name&gt;yarn.scheduler.maximum-allocation-mb&lt;/name&gt;
          &lt;value&gt;262144&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
          &lt;name&gt;yarn.nodemanager.resource.memory-mb&lt;/name&gt;
          &lt;value&gt;262144&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
          &lt;name&gt;yarn.scheduler.maximum-allocation-vcores&lt;/name&gt;
          &lt;value&gt;56&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
          &lt;name&gt;yarn.nodemanager.resource.cpu-vcores&lt;/name&gt;
          &lt;value&gt;56&lt;/value&gt;
  &lt;/property&gt;
  </code></pre>
  
  <p>Here is the yarn-site.xml file for the EMR cluster(which works)</p>
  
  <pre><code>&lt;configuration&gt;
  &lt;property&gt;
  &lt;name&gt;yarn.timeline-service.hostname&lt;/name&gt;
  &lt;value&gt;ip-172-31-63-120.ec2.internal&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
  &lt;name&gt;yarn.web-proxy.address&lt;/name&gt;
  &lt;value&gt;ip-172-31-63-120.ec2.internal:20888&lt;/value&gt;
  &lt;/property&gt; 
  
  &lt;property&gt;
  &lt;name&gt;yarn.resourcemanager.resource-tracker.address&lt;/name&gt;
  &lt;value&gt;ip-172-31-63-120.ec2.internal:8025&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
  &lt;name&gt;yarn.resourcemanager.address&lt;/name&gt;
  &lt;value&gt;ip-172-31-63-120.ec2.internal:8032&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
  &lt;name&gt;yarn.resourcemanager.scheduler.address&lt;/name&gt;
  &lt;value&gt;ip-172-31-63-120.ec2.internal:8030&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
  &lt;name&gt;yarn.log-aggregation-enable&lt;/name&gt;
  &lt;value&gt;true&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
  &lt;name&gt;yarn.log.server.url&lt;/name&gt;
  &lt;value&gt;http://ip-172-31-63-120.ec2.internal:19888/jobhistory/logs&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
  &lt;name&gt;yarn.dispatcher.exit-on-error&lt;/name&gt;
  &lt;value&gt;true&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
   &lt;name&gt;yarn.nodemanager.local-dirs&lt;/name&gt;
   &lt;value&gt;/mnt/yarn,/mnt1/yarn&lt;/value&gt;
  &lt;final&gt;true&lt;/final&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
  &lt;description&gt;Where to store container logs.&lt;/description&gt;
  &lt;name&gt;yarn.nodemanager.log-dirs&lt;/name&gt;
  &lt;value&gt;/var/log/hadoop-yarn/containers&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
  &lt;description&gt;Where to aggregate logs to.&lt;/description&gt;
  &lt;name&gt;yarn.nodemanager.remote-app-log-dir&lt;/name&gt;
  &lt;value&gt;/var/log/hadoop-yarn/apps&lt;/value&gt;
  &lt;/property&gt;
  
  
  &lt;property&gt;
   &lt;description&gt;Classpath for typical applications.&lt;/description&gt;
   &lt;name&gt;yarn.application.classpath&lt;/name&gt;
   &lt;value&gt;
      $HADOOP_CONF_DIR,
      $HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,
      $HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,
      $HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,
      $HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*,
      /usr/lib/hadoop-lzo/lib/*,
      /usr/share/aws/emr/emrfs/conf,
      /usr/share/aws/emr/emrfs/lib/*,
      /usr/share/aws/emr/emrfs/auxlib/*,
      /usr/share/aws/emr/lib/*,
      /usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar,
      /usr/share/aws/emr/goodies/lib/emr-hadoop-goodies.jar,
      /usr/lib/spark/yarn/lib/datanucleus-api-jdo.jar,
      /usr/lib/spark/yarn/lib/datanucleus-core.jar,
      /usr/lib/spark/yarn/lib/datanucleus-rdbms.jar,
      /usr/share/aws/emr/cloudwatch-sink/lib/*,
      /usr/share/aws/aws-java-sdk/*
   &lt;/value&gt;
   &lt;/property&gt;
  
  &lt;!-- The defaut setting (2.1) is silly. The virtual memory is not 
     a limiting factor on 64Bit systems, at least not a limiting  
      resource, so make it large, very large. --&gt;
  &lt;property&gt;
  &lt;name&gt;yarn.nodemanager.vmem-pmem-ratio&lt;/name&gt;
  &lt;value&gt;5&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
  &lt;name&gt;yarn.node-labels.enabled&lt;/name&gt;
  &lt;value&gt;true&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
  &lt;name&gt;yarn.node-labels.am.default-node-label-expression&lt;/name&gt;
  &lt;value&gt;CORE&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
  &lt;name&gt;yarn.node-labels.fs-store.root-dir&lt;/name&gt;
  &lt;value&gt;file:///mnt/var/lib/hadoop-yarn/nodelabels&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
  &lt;name&gt;yarn.node-labels.configuration-type&lt;/name&gt;
  &lt;value&gt;distributed&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
  &lt;name&gt;yarn.log-aggregation.enable-local-cleanup&lt;/name&gt;
  &lt;value&gt;false&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
  &lt;name&gt;yarn.nodemanager.address&lt;/name&gt;
  &lt;value&gt;${yarn.nodemanager.hostname}:8041&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
  &lt;name&gt;yarn.nodemanager.container-metrics.enable&lt;/name&gt;
  &lt;value&gt;false&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
  &lt;name&gt;yarn.nodemanager.recovery.enabled&lt;/name&gt;
  &lt;value&gt;true&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
  &lt;name&gt;yarn.nodemanager.recovery.supervised&lt;/name&gt;
  &lt;value&gt;true&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
  &lt;name&gt;yarn.resourcemanager.nodes.exclude-path&lt;/name&gt;
  &lt;value&gt;/emr/instance-controller/lib/yarn.nodes.exclude.xml&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
  &lt;name&gt;yarn.resourcemanager.webapp.cross-origin.enabled&lt;/name&gt;
  &lt;value&gt;true&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
  &lt;name&gt;yarn.scheduler.increment-allocation-mb&lt;/name&gt;
  &lt;value&gt;32&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
  &lt;name&gt;yarn.resourcemanager.nodemanagers.heartbeat-interval-ms&lt;/name&gt;
  &lt;value&gt;250&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
  &lt;name&gt;yarn.nodemanager.node-labels.provider&lt;/name&gt;
  &lt;value&gt;config&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
  &lt;name&gt;yarn.nodemanager.node-labels.provider.configured-node-partition&lt;/name&gt;
  &lt;value&gt;CORE&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
  &lt;name&gt;yarn.resourcemanager.system-metrics-publisher.enabled&lt;/name&gt;
  &lt;value&gt;true&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
  &lt;name&gt;yarn.timeline-service.http-cross-origin.enabled&lt;/name&gt;
  &lt;value&gt;true&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
  &lt;name&gt;yarn.resourcemanager.client.thread-count&lt;/name&gt;
  &lt;value&gt;64&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
  &lt;name&gt;yarn.nodemanager.resource.cpu-vcores&lt;/name&gt;
  &lt;value&gt;4&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
  &lt;name&gt;yarn.resourcemanager.resource-tracker.client.thread-count&lt;/name&gt;
  &lt;value&gt;64&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
  &lt;name&gt;yarn.nodemanager.container-manager.thread-count&lt;/name&gt;
  &lt;value&gt;64&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
  &lt;name&gt;yarn.resourcemanager.scheduler.client.thread-count&lt;/name&gt;
  &lt;value&gt;64&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
  &lt;name&gt;yarn.scheduler.maximum-allocation-mb&lt;/name&gt;
  &lt;value&gt;12288&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
  &lt;name&gt;yarn.nodemanager.localizer.client.thread-count&lt;/name&gt;
  &lt;value&gt;20&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
  &lt;name&gt;yarn.log-aggregation.retain-seconds&lt;/name&gt;
  &lt;value&gt;172800&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
  &lt;name&gt;yarn.nodemanager.localizer.fetch.thread-count&lt;/name&gt;
  &lt;value&gt;20&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
  &lt;name&gt;yarn.nodemanager.resource.memory-mb&lt;/name&gt;
  &lt;value&gt;12288&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
  &lt;name&gt;yarn.scheduler.maximum-allocation-vcores&lt;/name&gt;
  &lt;value&gt;128&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
  &lt;name&gt;yarn.resourcemanager.hostname&lt;/name&gt;
  &lt;value&gt;172.31.63.120&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
  &lt;name&gt;yarn.scheduler.minimum-allocation-mb&lt;/name&gt;
  &lt;value&gt;32&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
  &lt;name&gt;yarn.timeline-service.enabled&lt;/name&gt;
  &lt;value&gt;true&lt;/value&gt;
  &lt;/property&gt;
  </code></pre>
  
  <p></p>