title: How to sum values column wise for Array of Arrays in Scala?
body: <p>I am a newbie to Spark and Scala and trying to solve the below problem but couldn't. Please help me with this. Appreciate your help.</p>
  
  <p>The requirement is to sum the values column wise.
  The below code generates</p>
  
  <pre><code>val first = vlist.map(_.select("value"))
  first.map(_.show())
  </code></pre>
  
  <p>Output:</p>
  
  <pre><code>    first: Array[org.apache.spark.sql.DataFrame] = 
  Array([value: array&lt;double&gt;], [value: array&lt;double&gt;])
  
  +--------------------+
  |               value|
  +--------------------+
  |[-0.047363, 0.187...|
  |[0.043701, -0.114...|
  |[-0.006439, 0.031...|
  |[0.168945, 0.0639...|
  |[0.049805, 0.0664...|
  |[-0.054932, -0.11...|
  |[0.094727, -0.118...|
  |[0.136719, 0.1484...|
  |[-0.12793, 0.2812...|
  |[-0.071289, -0.07...|
  |[0.115234, -0.012...|
  |[0.253906, 0.0385...|
  |[-0.062988, 0.031...|
  |[0.110352, 0.2480...|
  |[0.042725, 0.2949...|
  |[-0.074219, 0.112...|
  |[0.072754, -0.092...|
  |[-0.063965, 0.058...|
  |[0.083496, -0.007...|
  |[0.043945, 0.1767...|
  +--------------------+
  only showing top 20 rows
  +--------------------+
  |               value|
  +--------------------+
  |[0.045654, -0.145...|
  |[0.053467, 0.0120...|
  |[0.033203, -0.089...|
  |[-0.08252, 0.0224...|
  |[0.182617, -0.044...|
  |[0.136719, 0.1484...|
  |[0.112793, -0.130...|
  |[0.096191, -0.028...|
  |[-0.007141, 0.004...|
  |[0.115234, -0.012...|
  |[0.130859, 0.0084...|
  |[-0.020874, 0.021...|
  |[-0.267578, 0.084...|
  |[-0.015015, 0.193...|
  |[0.036865, 0.0201...|
  |[0.205078, 0.0042...|
  |[-0.013733, -0.07...|
  |[0.175781, 0.2128...|
  |[-0.061279, -0.06...|
  |[0.058838, 0.3574...|
  +--------------------+
  </code></pre>
  
  <p>The next step should be the sum of all the values column wise. So, I should ideally end up with one row.</p>
  
  <p>I tried the below code:</p>
  
  <pre><code>first.toList.transpose.map(_.sum)
  </code></pre>
  
  <p>Output: </p>
  
  <pre><code>&lt;console&gt;:183: error: No implicit view available from   
   org.apache.spark.sql.DataFrame =&gt; scala.collection.GenTraversableOnce[B].
     first.toList.transpose.map(_.sum)
  </code></pre>
  
  <p>Also, I tried dividing the values into separate columns(took only 4 columns for testing purpose) and applied agg function which didn't work as well like below:</p>
  
  <pre><code>    var table = first
  for (i &lt;- 0 to 3) {
      table = table.map(_.withColumn("vec_" + i, $"value"(i)))
  }
  
  
  var inter = table.map(_.drop("value"))
  inter.map(_.show())
  
  var exprs = inter.map(_.columns.map(_ -&gt; "sum").toMap)
  inter.agg(exprs)
  </code></pre>
  
  <p>Output:</p>
  
  <pre><code>table: Array[org.apache.spark.sql.DataFrame] = 
  Array([value: array&lt;double&gt;], [value: array&lt;double&gt;])
  inter: Array[org.apache.spark.sql.DataFrame] = 
  Array([vec_0: double, 
  vec_1: double ... 2 more fields], 
  [vec_0: double,
   vec_1: double ... 2 more fields])
  +---------+---------+---------+---------+
  |    vec_0|    vec_1|    vec_2|    vec_3|
  +---------+---------+---------+---------+
  |-0.047363|   0.1875| 0.002258| 0.173828|
  | 0.043701|-0.114258| 0.067383|-0.060547|
  |-0.006439| 0.031982| 0.012878| 0.020264|
  | 0.168945| 0.063965|-0.084473| 0.173828|
  | 0.049805| 0.066406|  0.03833|  0.02356|
  |-0.054932|-0.117188| 0.027832| 0.074707|
  | 0.094727|-0.118652| 0.118164| 0.253906|
  | 0.136719| 0.148438| 0.114746| 0.069824|
  | -0.12793|  0.28125|  0.01532|-0.046631|
  |-0.071289| -0.07373| 0.199219|-0.069824|
  | 0.115234|-0.012512|-0.022949| 0.194336|
  | 0.253906| 0.038574|-0.030396| 0.248047|
  |-0.062988| 0.031494|-0.302734| 0.030396|
  | 0.110352| 0.248047| -0.00769|-0.031494|
  | 0.042725| 0.294922| 0.019653| 0.030884|
  |-0.074219| 0.112793| 0.094727| 0.071777|
  | 0.072754|-0.092773|-0.174805|-0.022583|
  |-0.063965| 0.058838| 0.086914| 0.320312|
  | 0.083496|-0.007294|-0.026489| -0.05957|
  | 0.043945| 0.176758| 0.094727|-0.083496|
  +---------+---------+---------+---------+
  only showing top 20 rows
  +---------+---------+---------+---------+
  |    vec_0|    vec_1|    vec_2|    vec_3|
  +---------+---------+---------+---------+
  | 0.045654|-0.145508|  0.15625| 0.166016|
  | 0.053467| 0.012024|  -0.0065| 0.008545|
  | 0.033203|-0.089844|-0.294922| 0.115234|
  | -0.08252| 0.022461|-0.149414| 0.099121|
  | 0.182617|-0.044922| 0.138672| 0.011658|
  | 0.136719| 0.148438| 0.114746| 0.069824|
  | 0.112793|-0.130859| 0.066895| 0.138672|
  | 0.096191|-0.028687|-0.108398| 0.145508|
  |-0.007141| 0.004486|  0.02063| 0.010803|
  | 0.115234|-0.012512|-0.022949| 0.194336|
  | 0.130859| 0.008423| 0.033447|-0.058838|
  |-0.020874| 0.021851|-0.083496|-0.072266|
  |-0.267578| 0.084961| 0.109863| 0.086914|
  |-0.015015| 0.193359| 0.014832|  0.07373|
  | 0.036865| 0.020142|  0.22168| 0.155273|
  | 0.205078| 0.004211| 0.084473| 0.091309|
  |-0.013733|-0.074219| 0.017334|-0.016968|
  | 0.175781| 0.212891|-0.071289| 0.084961|
  |-0.061279|-0.068359| 0.120117| 0.191406|
  | 0.058838| 0.357422| 0.128906|-0.162109|
  +---------+---------+---------+---------+
  only showing top 20 rows
  res4164: Array[Unit] = Array((), ())
  exprs: Array[scala.collection.immutable.Map[String,String]] = Array(Map(vec_0 -&gt; sum, vec_1 -&gt; sum, vec_2 -&gt; sum, vec_3 -&gt; sum), Map(vec_0 -&gt; sum, vec_1 -&gt; sum, vec_2 -&gt; sum, vec_3 -&gt; sum))
  &lt;console&gt;:189: error: value agg is not a member of Array[org.apache.spark.sql.DataFrame]
         inter.agg(exprs)
               ^
  </code></pre>
  
  <p>Please help me with this.</p>
  
  <p>I am sure there should be an easy way to do this. Thanks in advance. </p>
  
  <p>Adding Sample input and output. </p>
  
  <p>Sample Input :</p>
  
  <pre><code>first: Array[org.apache.spark.sql.DataFrame] = 
  Array([value: array&lt;double&gt;], [value: array&lt;double&gt;])
  
          value
          1,2,3,4,5,6 7,8
          1,2,3,4,5,6,7,8
  
          value 
          1,2,3,4,5,6 7,8
          1,2,3,4,5,6,7,8
  </code></pre>
  
  <p>Sample Output :</p>
  
  <pre><code>first: Array[org.apache.spark.sql.DataFrame] = 
  Array([value: array&lt;double&gt;], [value: array&lt;double&gt;])
  
          value
          2,4,6,8,10,14,16
  
          value
          2,4,6,8,10,14,16
  </code></pre>