title: Spark history server stops working in EMR when logs get large
body: <p>I am running a spark job on a 10 TB dataset using EMR.  I am using the Spark history server to monitor its progress.  However, when the logs get really large, the spark history server and the EMR UI both stop updating.  Is my EMR job still running or has it stopped working too?</p>
  <p>Furthermore, when the spark history server stops crashing, all my EC2 instances go from &gt; 75% CPU utilization to 0% utilization (they subsequently increase back to &lt;75%) and the EMR console shows 0 containers reserved and all memory freed (they also return to normal after).</p>
  <p>Has something happened to my EMR job?  Is there a way I can keep the Spark history server working when the logs get really large?</p>
  <p>Thanks.</p>