title: Dataproc Pyspark job only running on one node
body: <p>My problem is that my pyspark job is <em>not</em> running in parallel.</p>
  
  <p><strong>Code and data format:</strong><br>
  My PySpark looks something like this (simplified, obviously):</p>
  
  <pre><code>class TheThing:
      def __init__(self, dInputData, lDataInstance):
          # ...
      def does_the_thing(self):
          """About 0.01 seconds calculation time per row"""
          # ...
          return lProcessedData
  
  #contains input data pre-processed from other RDDs
  #done like this because one RDD cannot work with others inside its transformation
  #is about 20-40MB in size
  #everything in here loads and processes from BigQuery in about 7 minutes
  dInputData = {'dPreloadedData': dPreloadedData}
  
  #rddData contains about 3M rows
  #is about 200MB large in csv format
  #rddCalculated is about the same size as rddData
  rddCalculated = (
      rddData
          .map(
              lambda l, dInputData=dInputData: TheThing(dInputData, l).does_the_thing()
          )
  )
  
  llCalculated = rddCalculated.collect()
  #save as csv, export to storage
  </code></pre>
  
  <p><strong>Running on Dataproc cluster:</strong><br>
  Cluster is created via the <a href="https://console.cloud.google.com/dataproc/clusters" rel="nofollow">Dataproc UI</a>.<br>
  Job is executed like this:<br>
  <code>gcloud --project &lt;project&gt; dataproc jobs submit pyspark --cluster &lt;cluster_name&gt; &lt;script.py&gt;</code>  </p>
  
  <p>I observed the job status via the UI, <a href="https://cloud.google.com/dataproc/cluster-web-interfaces#step_2_-_connect_with_your_web_browser" rel="nofollow">started like this</a>.  Browsing through it I noticed that only one (seemingly random) of my worker nodes was doing anything.  All others were completely idle.</p>
  
  <p>Whole point of PySpark is to run this thing in parallel, and is obviously not the case.  I've run this data in all sorts of cluster configurations, the last one being massive, which is when I noticed it's singular-node use.  And hence why my jobs take too very long to complete, and time seems independent of cluster size.  </p>
  
  <p>All tests with smaller datasets pass without problems on my local machine and on the cluster.  I really just need to upscale. </p>
  
  <p><strong>EDIT</strong><br>
  I changed<br>
  <code>llCalculated = rddCalculated.collect()</code><br>
  <code>#... save to csv and export</code><br>
  to<br>
  <code>rddCalculated.saveAsTextFile("gs://storage-bucket/results")</code>  </p>
  
  <p>and only one node is still doing the work.</p>