title: What hadoop configuration setting determines number of nodes available in spark?
body: <p>Not much experience with spark and trying to determine amount of available memory, number of executors, and nodes for a submitted spark job. Code just looks like...</p>
  
  <pre class="lang-py prettyprint-override"><code>from pyspark.sql import SparkSession
  from pyspark.sql.functions import *
  import time
  
  sparkSession = SparkSession.builder.appName("node_count_test").getOrCreate()
  sparkSession._jsc.sc().setLogLevel("WARN")
  
  # see https://stackoverflow.com/a/52516704/8236733 
  print("Giving some time to let session start in earnest...")
  time.sleep(15)
  print("...done")
  
  print("\n\nYou are using %d nodes in this session\n\n" % sparkSession._jsc.sc().getExecutorMemoryStatus().keySet().size())
  </code></pre>
  
  <p>and the output is...</p>
  
  <pre class="lang-sh prettyprint-override"><code>Giving some time to let session start in earnest...
  ...done
  
  
  You are using 3 nodes in this session
  </code></pre>
  
  <p>I would think this number should be the number of <em>data nodes</em> in the cluster, which I can see in ambari is 4, so I would think the output above would be 4. Can anyone tell me what determines the number of available nodes in spark or how I can scope into this further?</p>