title: Calculate the standard deviation of grouped data in a Spark DataFrame
body: <p>I have user logs that I have taken from a csv and converted into a DataFrame in order to leverage the SparkSQL querying features.  A single user will create numerous entries per hour, and I would like to gather some basic statistical information for each user; really just the count of the user instances, the average, and the standard deviation of numerous columns.  I was able to quickly get the mean and count information by using groupBy($"user") and the aggregator with SparkSQL functions for count and avg:</p>
  
  <pre><code>val meanData = selectedData.groupBy($"user").agg(count($"logOn"),
  avg($"transaction"), avg($"submit"), avg($"submitsPerHour"), avg($"replies"),
  avg($"repliesPerHour"), avg($"duration"))
  </code></pre>
  
  <p>However, I cannot seem to find an equally elegant way to calculate the standard deviation.  So far I can only calculate it by mapping a string, double pair and use StatCounter().stdev utility:</p>
  
  <pre><code>val stdevduration = duration.groupByKey().mapValues(value =&gt;
  org.apache.spark.util.StatCounter(value).stdev)
  </code></pre>
  
  <p>This returns an RDD however, and I would like to try and keep it all in a DataFrame for further queries to be possible on the returned data.</p>