title: EMR 5.13: Spark 2.3.0 UI shows Executors remain alive
body: <p>Ever since I've upgraded to <code>EMR 5.13</code>, I've been seeing strange metrics on the <code>Spark</code> &amp; <code>YARN</code> <em>UIs</em>.</p>
  
  <p>In this particular instance:</p>
  
  <ul>
  <li><code>YARN</code> showed that process completed
  <a href="https://i.stack.imgur.com/1VDR2.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/1VDR2.jpg" alt="YARN reports task completed"></a></li>
  <li><code>Ganglia</code> shows that <em>cluster</em> has been <em>idle</em> since completion of last (118th) job</li>
  <li><code>Spark</code> UI also tells that all my 118 tasks have been completed
    <a href="https://i.stack.imgur.com/eoT3q.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/eoT3q.png" alt="All 118 jobs were completed"></a></li>
  </ul>
  
  <p>Even so, <code>Spark</code> UI reports that all <strong><code>Executor</code>s are alive</strong>, long (over 1 hr at time of writing) after the last <em>job</em> was completed.
  <a href="https://i.stack.imgur.com/6eZ6l.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/6eZ6l.jpg" alt="Executors are still alive with active tasks"></a></p>
  
  <hr>
  
  <p>Could this be a UI glitch or there's something else going on?</p>
  
  <p>Frameworks / Platform:</p>
  
  <ul>
  <li><code>EMR 5.13</code></li>
  <li><code>Spark 2.3.0</code></li>
  <li><code>Hive 2.3.2</code></li>
  <li><code>Hadoop: Amazon 2.8.3</code></li>
  </ul>