title: No stages are running, but numRunningTasks != 0
body: <p>My task was completed and I got the expected result of counting an RDD. I am running an interactive PySpark shell. I am trying to understand what does this warning mean:</p>
  
  <blockquote>
    <p>WARN ExecutorAllocationManager: No stages are running, but
    numRunningTasks != 0</p>
  </blockquote>
  
  <p>From Spark's internal <a href="https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala" rel="nofollow noreferrer">code</a> I found this:</p>
  
  <pre><code>    // If this is the last stage with pending tasks, mark the scheduler queue as empty
      // This is needed in case the stage is aborted for any reason
      if (stageIdToNumTasks.isEmpty) {
        allocationManager.onSchedulerQueueEmpty()
        if (numRunningTasks != 0) {
          logWarning("No stages are running, but numRunningTasks != 0")
          numRunningTasks = 0
        }
      }
  </code></pre>
  
  <p>Can someone explain please?</p>
  
  <hr>
  
  <p>I am talking about the Task with Id 0.</p>
  
  <p><a href="https://i.stack.imgur.com/iTdec.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/iTdec.png" alt="enter image description here"></a></p>
  
  <hr>
  
  <p>I can report that experience this behavior with MLlib of Spark, with <code>KMeans()</code>, where <a href="https://stackoverflow.com/questions/38986395/sparkkmeans-calls-takesample-twice">the one of the two samples</a> is said to be completed with less tasks. I am not sure if the job will fail or not yet..</p>
  
  <pre><code>2  takeSample at KMeans.scala:355 2016/08/27 21:39:04   7 s 1/1 9600/9600
  1  takeSample at KMeans.scala:355 2016/08/27 21:38:57   6 s 1/1 6608/9600
  </code></pre>
  
  <hr>
  
  <p>The input set is 100m points, of 256 dimensions.</p>
  
  <p>Some of the parameters to PySpark: master is yarn, mode is cluster, </p>
  
  <pre><code>spark.dynamicAllocation.enabled             false
  # Better serializer - https://spark.apache.org/docs/latest/tuning.html#data-serialization
  spark.serializer                            org.apache.spark.serializer.KryoSerializer
  spark.kryoserializer.buffer.max             2000m
  
  # Bigger PermGen space, use 4 byte pointers (since we have &lt; 32GB of memory)
  spark.executor.extraJavaOptions             -XX:MaxPermSize=512m -XX:+UseCompressedOops
  
  # More memory overhead
  spark.yarn.executor.memoryOverhead          4096
  spark.yarn.driver.memoryOverhead            8192
  
  spark.executor.cores                        8
  spark.executor.memory                       8G
  
  spark.driver.cores                          8
  spark.driver.memory                         8G
  spark.driver.maxResultSize                  4G
  </code></pre>