title: How convert Spark dataframe column from Array[Int] to linalg.Vector?
body: <p>I have a dataframe, df, that looks like this:</p>
  
  <pre><code>+--------+--------------------+
  | user_id|        is_following|
  +--------+--------------------+
  |       1|[2, 3, 4, 5, 6, 7]  |
  |       2|[20, 30, 40, 50]    |
  +--------+--------------------+
  </code></pre>
  
  <p>I can confirm this has the schema:</p>
  
  <pre><code>root
   |-- user_id: integer (nullable = true)
   |-- is_following: array (nullable = true)
   |    |-- element: integer (containsNull = true)
  </code></pre>
  
  <p>I would like to use Spark's ML routines such as LDA to do some machine learning on this, requiring me to convert the <code>is_following</code> column to a <code>linalg.Vector</code> (not a Scala vector).  When I try to do this via</p>
  
  <pre><code>import org.apache.spark.ml.feature.VectorAssembler
  import org.apache.spark.ml.linalg.Vectors
  
  val assembler = new VectorAssembler().setInputCols(Array("is_following")).setOutputCol("features")
  val output = assembler.transform(df)
  </code></pre>
  
  <p>I then get the following error:</p>
  
  <pre><code>java.lang.IllegalArgumentException: Data type ArrayType(IntegerType,true) is not supported.
  </code></pre>
  
  <p>If I am interpreting that correctly, I take away from it that I need to convert types here from integer to something else. (Double?  String?)</p>
  
  <p>My question is what is the best way to convert this array to something that will properly vectorize for the ML pipeline? </p>
  
  <p>EDIT: If it helps, I don't have to structure the dataframe this way. I could instead have it be:</p>
  
  <pre><code>+--------+------------+
  | user_id|is_following|
  +--------+------------+
  |       1|           2|
  |       1|           3|
  |       1|           4|
  |       1|           5|
  |       1|           6|
  |       1|           7|
  |       2|          20|
  |     ...|         ...|
  +--------+------------+
  </code></pre>