title: Creating a Random Feature Array in Spark DataFrames
body: <p>When creating an <code>ALS</code> model, we can extract a <code>userFactors</code> DataFrame and an <code>itemFactors</code> DataFrame. These DataFrames contain a column with an Array.</p>
  
  <p>I would like to generate some random data and union it to the <code>userFactors</code> DataFrame.</p>
  
  <p>Here is my code:</p>
  
  <pre><code> val df1: DataFrame  = Seq((123, 456, 4.0), (123, 789, 5.0), (234, 456, 4.5), (234, 789, 1.0)).toDF("user", "item", "rating")
  val model1 = (new ALS()
   .setImplicitPrefs(true)
   .fit(df1))
  
  val iF = model1.itemFactors
  val uF = model1.userFactors
  </code></pre>
  
  <p>I then create a random DataFrame using a <code>VectorAssembler</code> with this function:</p>
  
  <pre><code>def makeNew(df: DataFrame, rank: Int): DataFrame = {
      var df_dummy = df
      var i: Int = 0
      var inputCols: Array[String] = Array()
      for (i &lt;- 0 to rank) {
         df_dummy = df_dummy.withColumn("feature".concat(i.toString), rand())
         inputCols = inputCols :+ "feature".concat(i.toString)
        }
      val assembler = new VectorAssembler()
        .setInputCols(inputCols)
        .setOutputCol("userFeatures")
      val output = assembler.transform(df_dummy)
      output.select("user", "userFeatures")
    }
  </code></pre>
  
  <p>I then create the DataFrame with new user IDs and add the random vectors and bias:</p>
  
  <pre><code>val usersDf: DataFrame = Seq(567), (678)).toDF("user")
  var usersFactorsNew: DataFrame = makeNew(usersDf, 20)
  </code></pre>
  
  <p>The problem arises when I union the two DataFrames. </p>
  
  <p><code>usersFactorsNew.union(uF)</code> produces the error:</p>
  
  <pre><code> org.apache.spark.sql.AnalysisException: Union can only be performed on tables with the compatible column types. struct&lt;type:tinyint,size:int,indices:array&lt;int&gt;,values:array&lt;double&gt;&gt; &lt;&gt; array&lt;float&gt; at the second column of the second table;;
  </code></pre>
  
  <p>If I print the schema, the <code>uF</code> DataFrame has a feature vector of type <code>Array[Float]</code> and the <code>usersFactorsNew</code> DataFrame as a feature vector of type <code>Vector</code>.</p>
  
  <p>My question is how to change the type of the <code>Vector</code> to an Array in order to perform the union.</p>
  
  <p>I tried writing this <code>udf</code> with little success:</p>
  
  <pre><code>val toArr: org.apache.spark.ml.linalg.Vector =&gt; Array[Double] = _.toArray
  val toArrUdf = udf(toArr)
  </code></pre>
  
  <p>Perhaps the <code>VectorAssembler</code> is not the best option for this task. However, at the moment, it is the only option I have found. I would love to get some recommendations for something better.</p>