title: Column manipulations in Spark Scala
body: <p>I am learning to work with Apache Spark (Scala) and still figuring out how things work out here</p>
  <p>I am trying to achieve a simple task of</p>
  <ol>
  <li>Finding max of column</li>
  <li>Subtract each value of the column from this max and create a new column</li>
  </ol>
  <p>The code I am using is</p>
  <pre><code>import org.apache.spark.sql.functions._
  val training = sqlContext.createDataFrame(Seq(
    (10),
    (13),
    (14),
    (21)
  )).toDF(&quot;Values&quot;)
  
  val training_max = training.withColumn(&quot;Val_Max&quot;,training.groupBy().agg(max(&quot;Values&quot;))
  val training_max_sub = training_max.withColumn(&quot;Subs&quot;,training_max.groupBy().agg(col(&quot;Val_Max&quot;)-col(&quot;Values) ))
  </code></pre>
  <p>However I am getting a lot of errors. I am more or less fluent in R and had I been doing the same task my code would have been:</p>
  <pre><code>library(dplyr)
  new_data &lt;- training %&gt;%
      mutate(Subs= max(Values) - Values)
  </code></pre>