title: Merge two columns of type Array[string] into a new Array[string] column
body: <p>I have two columns in a Spark SQL <code>DataFrame</code> with each entry in either column as an array of strings. </p>
  
  <pre><code>val  ngramDataFrame = Seq(
    (Seq("curious", "bought", "20"), Seq("iwa", "was", "asj"))
  ).toDF("filtered_words", "ngrams_array")
  </code></pre>
  
  <p>I want to merge the arrays in each row to make a single array in a new column. My code is as follows:</p>
  
  <pre><code>def concat_array(firstarray: Array[String], 
                   secondarray: Array[String]) : Array[String] = 
                                       { (firstarray ++ secondarray).toArray }
  val concatUDF = udf(concat_array _)
  val concatFrame = ngramDataFrame.withColumn("full_array", concatUDF($"filtered_words", $"ngrams_array"))
  </code></pre>
  
  <p>I can successfully use the <code>concat_array</code> function on two arrays. However when I run the above code, I get the following exception:</p>
  
  <blockquote>
    <p>org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 16.0 failed 1 times, most recent failure: Lost task 0.0 in stage 16.0 (TID 12, localhost): org.apache.spark.SparkException: Failed to execute user defined function(anonfun$1: (array, array) => array) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source) at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:370) at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:389) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408) at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:125) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47) at org.apache.spark.scheduler.Task.run(Task.scala:86) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.ClassCastException: scala.collection.mutable.WrappedArray$ofRef cannot be cast to [Ljava.lang.String; at $line80.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$1.apply(:76) ... 13 more Driver stacktrace:</p>
  </blockquote>