title: How to increase Spark log level - Shutdown Hook Called
body: <p>I am getting this error and I was told I can increase log level of spark to investigate what's wrong. I am not sure how to do that. Please try to provide a step-by-step guide to that as I am very new to Spark.</p>
  <p><a href="https://i.stack.imgur.com/dvAtz.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/dvAtz.png" alt="enter image description here" /></a></p>
  <p>So far I have tried to edit the log4j.properties file but I got &quot;permission denied&quot;. I don't think the admin would allow me to edit it as well.</p>
  <p>Below is the content of the step1.sh file I was trying to run:</p>
  <p><a href="https://i.stack.imgur.com/8Tplr.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/8Tplr.png" alt="" /></a></p>
  <p>Thank you for the help.</p>
  <p><strong><strong>Update:</strong></strong></p>
  <p>So I checked the yarn log of one of the applications that failed.
  What does this indicate?
  <a href="https://i.stack.imgur.com/iLvxm.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/iLvxm.png" alt="enter image description here" /></a></p>