title: How the Number of partitions and Number of concurrent tasks in spark calculated
body: <p>I have a cluster with 4 nodes (each with 16 cores) using Spark 1.0.1.</p>
  
  <p>I have an RDD which I've repartitioned so it has 200 partitions (hoping to increase the parallelism).</p>
  
  <p>When I do a transformation (such as filter) on this RDD, I can't  seem to get more than 64 tasks (my total number of cores across the 4 nodes) going at one point in time.  By tasks, I mean the number of tasks that appear under the Application Spark UI.  I tried explicitly setting the spark.default.parallelism to 128 (hoping I would get 128 tasks concurrently running) and verified this in the Application UI for the running application but this had no effect.  Perhaps, this is ignored for a 'filter' and the default is the total number of cores available.</p>
  
  <p>I'm fairly new with Spark so maybe I'm just missing or misunderstanding something fundamental.  Any help would be appreciated.</p>