title: Spark dataframe saveAsTable is using a single task
body: <p>We have a pipeline for which the initial stages are properly scalable - using several dozen workers apiece.</p>
  
  <p>One of the last stages is </p>
  
  <pre><code>dataFrame.write.format(outFormat).mode(saveMode).
  partitionBy(partColVals.map(_._1): _*).saveAsTable(tname)
  </code></pre>
  
  <p>For this stage we end up with a <em>single</em> worker. This clearly does not work for us - in fact the worker runs out of disk space - on top of being very slow.</p>
  
  <p><a href="https://i.stack.imgur.com/Pf8m5.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Pf8m5.jpg" alt="enter image description here"></a></p>
  
  <p>Why would that command end up running on a single worker/single task only?</p>
  
  <p><strong>Update</strong>  The output format was <code>parquet</code>. The number of partition columns did not affect the result (tried one column as well as several columns).</p>
  
  <p><strong>Another update</strong>  <em>None</em> of the following conditions (as posited by an answer below) held:</p>
  
  <ul>
  <li><code>coalesce</code> or <code>partitionBy</code> statements</li>
  <li><code>window</code> / analytic functions</li>
  <li><code>Dataset.limit</code></li>
  <li><code>sql.shuffle.partitions</code></li>
  </ul>