title: How can I tell if my spark job is progressing?
body: <p>I have a spark job running on <code>YARN</code> and it appears to just hang and not be doing any computation.  </p>
  
  <p>Here's what yarn says when I do <code>yarn application -status &lt;APPLICATIOM ID&gt;</code>: </p>
  
  <pre><code>Application Report : 
  Application-Id : applicationID
  Application-Name : test app
  Application-Type : SPARK
  User : ec2-user
  Queue : default
  Start-Time : 1491005660004
  Finish-Time : 0
  Progress : 10%
  State : RUNNING
  Final-State : UNDEFINED
  Tracking-URL : http://&lt;ip&gt;:4040
  RPC Port : 0
  AM Host : &lt;host ip&gt;
  Aggregate Resource Allocation : 36343926 MB-seconds, 9818 vcore-seconds
  Log Aggregation Status : NOT_START
  Diagnostics : 
  </code></pre>
  
  <p>And, when I check the <code>yarn application -list</code> it says that it is <code>RUNNING</code>.  But I'm not sure I trust that.  When I go to the spark webUI, I see only one stage the entire few hours I've been running it:</p>
  
  <p><a href="https://i.stack.imgur.com/c65H1.png" rel="noreferrer"><img src="https://i.stack.imgur.com/c65H1.png" alt="web UI"></a></p>
  
  <p>Also, when I click on the "Stages" tab, I see nothing running:</p>
  
  <p><a href="https://i.stack.imgur.com/3GRg0.png" rel="noreferrer"><img src="https://i.stack.imgur.com/3GRg0.png" alt="Stages tab"></a></p>
  
  <p>How do ensure that my application is actually running and that <code>YARN</code> is not lying to me?  </p>
  
  <p>I would actually prefer for this to throw an error rather than keep me waiting to see if the job is actaully runing.  How do I do that?</p>