title: Spark Parallelism in Standalone Mode
body: <p>I'm trying to run spark in standalone mode in my system. The current specification of my system is 8 cores and 32 Gb memory. Base on <a href="https://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-2/" rel="noreferrer">this article</a> I calculate the spark configurations as the following:</p>
  
  <pre><code>spark.driver.memory 2g
  spark.executor.cores 3
  spark.executor.instances 2
  spark.executor.memory 20g
  maximizeResourceAllocation TRUE
  </code></pre>
  
  <p>I created spark context in my jupyter notebook like this and was checking the parallelism level, by this</p>
  
  <pre><code>sc = SparkContext()
  sc.defaultParallelism
  </code></pre>
  
  <p>The default parallelism is giving me 8. My question is why it's giving me 8 even though I mentioned 2 cores? If it's not giving me the actual parallelism of my system, then how to get the actual level of parallelism?</p>
  
  <p>Thank you!</p>