title: Normalize values of multiple columns in Spark DataFrame, using only DataFrame API
body: <p>I am trying to normalize the values of multiple columns in a spark dataframe, by subtracting the mean and dividing by the stddev of each column. Here's the code I have so far:</p>
  
  <pre class="lang-py prettyprint-override"><code>from pyspark.sql import Row
  from pyspark.sql.functions import stddev_pop, avg
  
  df = spark.createDataFrame([Row(A=1, B=6), Row(A=2, B=7), Row(A=3, B=8),
                              Row(A=4, B=9), Row(A=5, B=10)])
  
  exprs = [x - (avg(x)) / stddev_pop(x) for x in df.columns]    
  df.select(exprs).show() 
  </code></pre>
  
  <p>Which gives me the result:</p>
  
  <pre><code>+------------------------------+------------------------------+
  |(A - (avg(A) / stddev_pop(A)))|(B - (avg(B) / stddev_pop(B)))|
  +------------------------------+------------------------------+
  |                          null|                          null|
  +------------------------------+------------------------------+
  </code></pre>
  
  <p>Where I'm hoping for:</p>
  
  <pre><code>+------------------------------+------------------------------+
  |(A - (avg(A) / stddev_pop(A)))|(B - (avg(B) / stddev_pop(B)))|
  +------------------------------+------------------------------+
  |                  -1.414213562|                  -1.414213562|
  |                  -0.707106781|                  -0.707106781|
  |                             0|                             0|
  |                   0.707106781|                   0.707106781|
  |                   1.414213562|                   1.414213562|
  +------------------------------+------------------------------+
  </code></pre>
  
  <p>I believe I can do this with the <a href="http://spark.apache.org/docs/latest/mllib-feature-extraction.html#standardscaler" rel="nofollow">StandardScaler</a> class from mllib, but I'd prefer to do this using only the dataframe API if possible - if only as a learning exercise.</p>