title: Why does Spark (on Google Dataproc) not use all vcores?
body: <p>I'm running a spark job on a Google DataProc cluster. But looks like Spark is not using all the <code>vcores</code> available in the cluster as you can see below</p>
  
  <p><a href="https://i.stack.imgur.com/oXiH0.png" rel="noreferrer"><img src="https://i.stack.imgur.com/oXiH0.png" alt="Cluster usage"></a></p>
  
  <p>Based on some other questions like <a href="https://stackoverflow.com/questions/38368985/spark-on-yarn-too-less-vcores-used">this</a> and <a href="https://stackoverflow.com/questions/33458063/why-does-vcore-always-equal-the-number-of-nodes-in-spark-on-yarn">this</a>, i have setup the cluster to use <code>DominantResourceCalculator</code> to consider both vcpus and memory for resource allocation</p>
  
  <pre><code>gcloud dataproc clusters create cluster_name --bucket="profiling-
  job-default" \
  --zone=europe-west1-c \
  --master-boot-disk-size=500GB \
  --worker-boot-disk-size=500GB \
  --master-machine-type=n1-standard-16 \
  --num-workers=10 \
  --worker-machine-type=n1-standard-16 \
  --initialization-actions gs://custom_init_gcp.sh \
  --metadata MINICONDA_VARIANT=2 \
  --properties=^--^yarn:yarn.scheduler.capacity.resource-calculator=org.apache.hadoop.yarn.util.resource.DominantResourceCalculator
  </code></pre>
  
  <p>But when i submit my job with custom spark flags, looks like YARN doesn't respect these custom parameters and defaults to using memory as the yardstick for resource calculation</p>
  
  <pre><code>gcloud dataproc jobs submit pyspark --cluster cluster_name \
  --properties spark.sql.broadcastTimeout=900,spark.network.timeout=800\
  ,yarn.scheduler.capacity.resource-calculator=org.apache.hadoop.yarn.util.resource.DominantResourceCalculator\
  ,spark.dynamicAllocation.enabled=true\
  ,spark.executor.instances=10\
  ,spark.executor.cores=14\
  ,spark.executor.memory=15g\
  ,spark.driver.memory=50g \
  src/my_python_file.py 
  </code></pre>
  
  <p>Can help somebody figure out what's going on here?</p>