title: Apply a transformation to multiple columns pyspark dataframe
body: <p>Suppose I have the following spark-dataframe:</p>
  
  <pre class="lang-python prettyprint-override"><code>+-----+-------+
  | word|  label|
  +-----+-------+
  |  red|  color|
  |  red|  color|
  | blue|  color|
  | blue|feeling|
  |happy|feeling|
  +-----+-------+
  </code></pre>
  
  <p>Which can be created using the following code:</p>
  
  <pre class="lang-python prettyprint-override"><code>sample_df = spark.createDataFrame([
          ('red', 'color'),
          ('red', 'color'),
          ('blue', 'color'),
          ('blue', 'feeling'),
          ('happy', 'feeling')
      ],
      ('word', 'label')
  )
  </code></pre>
  
  <p>I can perform a <code>groupBy()</code> to get the counts of each word-label pair:</p>
  
  <pre class="lang-python prettyprint-override"><code>sample_df = sample_df.groupBy('word', 'label').count()
  #+-----+-------+-----+
  #| word|  label|count|
  #+-----+-------+-----+
  #| blue|  color|    1|
  #| blue|feeling|    1|
  #|  red|  color|    2|
  #|happy|feeling|    1|
  #+-----+-------+-----+
  </code></pre>
  
  <p>And then <code>pivot()</code> and <code>sum()</code> to get the label counts as columns:</p>
  
  <pre class="lang-python prettyprint-override"><code>import pyspark.sql.functions as f
  sample_df = sample_df.groupBy('word').pivot('label').agg(f.sum('count')).na.fill(0)
  #+-----+-----+-------+
  #| word|color|feeling|
  #+-----+-----+-------+
  #|  red|    2|      0|
  #|happy|    0|      1|
  #| blue|    1|      1|
  #+-----+-----+-------+
  </code></pre>
  
  <p>What is the best way to transform this <code>dataframe</code> such that each row is divided by the total for that row? </p>
  
  <pre class="lang-python prettyprint-override"><code># Desired output
  +-----+-----+-------+
  | word|color|feeling|
  +-----+-----+-------+
  |  red|  1.0|    0.0|
  |happy|  0.0|    1.0|
  | blue|  0.5|    0.5|
  +-----+-----+-------+
  </code></pre>
  
  <p>One way to achieve this result is to use <code>__builtin__.sum</code> (NOT <code>pyspark.sql.functions.sum</code>) to get the row-wise sum and then call <code>withColumn()</code> for each label:</p>
  
  <pre class="lang-python prettyprint-override"><code>labels = ['color', 'feeling']
  
  sample_df.withColumn('total', sum([f.col(x) for x in labels]))\
      .withColumn('color', f.col('color')/f.col('total'))\
      .withColumn('feeling', f.col('feeling')/f.col('total'))\
      .select('word', 'color', 'feeling')\
      .show()
  </code></pre>
  
  <p>But there has to be a better way than enumerating each of the possible columns.</p>
  
  <p>More generally, my question is:</p>
  
  <p><strong>How can I apply an arbitrary transformation, that is a function of the current row, to multiple columns simultaneously?</strong></p>