title: Convert Spark RDD to dataset
body: <p>I am trying to make a kmean clustering after some text mining but I can't find how to convert the result of ParseWikipedia.termDocumentMatrix in a dataset required by the kmean.fit method</p>
  
  <pre><code>scala&gt; val (termDocMatrix, termIds, docIds, idfs) = ParseWikipedia.termDocumentMatrix(lemmas, stopWords, numTerms, sc)
  scala&gt; val kmeans = new KMeans().setK(5).setMaxIter(200).setSeed(1L)
  scala&gt; termDocMatrix.take(1)
  res24: Array[org.apache.spark.mllib.linalg.Vector] = Array((1000,[32,166,200,223,577,645,685,873,926],[0.18132966949934762,0.3777537726516676,0.3178848913768969,0.43380819546465704,0.30604090845847254,0.46007361524957147,0.2076406414508386,0.2995665853335863,0.1742843713808876]))
  
  scala&gt; val modele = kmeans.fit(termDocMatrix)
  &lt;console&gt;:66: error: type mismatch;
   found   : org.apache.spark.rdd.RDD[org.apache.spark.mllib.linalg.Vector]
   required: org.apache.spark.sql.Dataset[_]
         val modele = kmeans.fit(termDocMatrix)
  </code></pre>
  
  <p>I tried some conversions but i always have errors</p>
  
  <pre><code>scala&gt; import spark.implicits._
  import spark.implicits._
  
  scala&gt; val ss=org.apache.spark.sql.SparkSession.builder().getOrCreate()
  scala&gt; ss.createDataset(termDocMatrix)
  &lt;console&gt;:67: error: Unable to find encoder for type stored in a Dataset.  Primitive types (Int, String, etc) and Product types (case classes) are supported by importing spark.implicits._  Support for serializing other types will be added in future releases.
     ss.createDataset(termDocMatrix)
  </code></pre>
  
  <p>and others (with a expected result as it was not to datasets)</p>
  
  <pre><code>val termDocRows = termDocMatrix.map(org.apache.spark.sql.Row(_))
  val schemaVecteurs = StructType(Seq(StructField("features", VectorType, true)))
  val termDocVectors = spark.createDataFrame(termDocRows, schemaVecteurs)
  val termDocMatrixDense = termDocMatrix.map(e =&gt; e.toDense)
  </code></pre>
  
  <p>(and try to kmeans.fit each of them). The only one that gives a different error is termDocVectors</p>
  
  <pre><code>val modele = kmeans.fit(termDocVectors)
  18/01/05 01:14:52 ERROR Executor: Exception in task 0.0 in stage 560.0 (TID 1682)
  java.lang.RuntimeException: Error while encoding: java.lang.RuntimeException: org.apache.spark.mllib.linalg.SparseVector is not a valid external type for schema of vector
  if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else newInstance(class org.apache.spark.ml.linalg.VectorUDT).serialize AS features#75
      at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:290)
  </code></pre>
  
  <p>Do someone has a clue ?
  Thanks for your help</p>
  
  <p><strong>In addition after testing the clues provided :</strong></p>
  
  <p><strong>Where can I apply toDS ?</strong></p>
  
  <pre><code>scala&gt; termDocMatrix.toDS
  &lt;console&gt;:69: error: value toDS is not a member of org.apache.spark.rdd.RDD[org.apache.spark.mllib.linalg.Vector]
     termDocMatrix.toDS
  </code></pre>
  
  <p><strong>With Tuple...</strong><br/>
  I still have an error (different, this time)</p>
  
  <pre><code>val ds = spark.createDataset(termDocMatrix.map(Tuple1.apply)).withColumnRenamed("_1", "features")
  ds: org.apache.spark.sql.DataFrame = [features: vector]
  scala&gt; val modele = kmeans.fit(ds)
  java.lang.IllegalArgumentException: requirement failed: Column features must be of type org.apache.spark.ml.linalg.VectorUDT@3bfc3ba7 but was actually org.apache.spark.mllib.linalg.VectorUDT@f71b0bce.
  </code></pre>
  
  <p>The initial problem seems to be solved. Now i'm facing a new one with the fact that i computeSVD from a mllib.Rowmatrix and kmeans seems to wait ml vectors. I just have to find how to compute a SVD in the ml packages...</p>