title: Getting "Container killed by YARN for exceeding memory limits." error in AWS Glue Notebook
body: <p>We are running an ETL script on Glue Notebook and facing the following error. Based on the error, we have set the configuration <em>spark.yarn.executor.memoryOverhead</em> to 10 GB. We couldn't disable <em>yarn.nodemanager.vmem-check-enabled</em> as we couldn't access yarn xml configuration file in AWS Glue Endpoint. Error log is as follows:</p>
  <pre><code>An error occurred while calling o1141.showString.
  : org.apache.spark.SparkException: Job aborted due to stage failure: Task 171 in stage 
  
  35.0 failed 4 times, most recent failure: Lost task 171.3 in stage 35.0 
  (TID 8929, ip_addr.ap-southeast-1.compute.internal, executor 13): 
  ExecutorLostFailure (executor 13 exited caused by one of the running tasks) 
  Reason: Container killed by YARN for exceeding memory limits.  
  5.8 GB of 5.5 GB physical memory used. 
  Consider boosting spark.yarn.executor.memoryOverhead or 
  disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714.
  </code></pre>
  <p>Can anyone suggest how we can fix this issue?</p>