title: How to convert spark DataFrame to RDD mllib LabeledPoints?
body: <p>I tried to apply PCA to my data and then apply RandomForest to the transformed data. However, PCA.transform(data) gave me a DataFrame but I need a mllib LabeledPoints to feed my RandomForest. How can I do that?
  My code:</p>
  
  <pre><code>    import org.apache.spark.mllib.util.MLUtils
      import org.apache.spark.{SparkConf, SparkContext}
      import org.apache.spark.mllib.tree.RandomForest
      import org.apache.spark.mllib.tree.model.RandomForestModel
      import org.apache.spark.ml.feature.PCA
      import org.apache.spark.mllib.regression.LabeledPoint
      import org.apache.spark.mllib.linalg.Vectors
  
  
      val dataset = MLUtils.loadLibSVMFile(sc, "data/mnist/mnist.bz2")
  
      val splits = dataset.randomSplit(Array(0.7, 0.3))
  
      val (trainingData, testData) = (splits(0), splits(1))
  
      val trainingDf = trainingData.toDF()
  
      val pca = new PCA()
      .setInputCol("features")
      .setOutputCol("pcaFeatures")
      .setK(100)
      .fit(trainingDf)
  
      val pcaTrainingData = pca.transform(trainingDf)
  
      val numClasses = 10
      val categoricalFeaturesInfo = Map[Int, Int]()
      val numTrees = 10 // Use more in practice.
      val featureSubsetStrategy = "auto" // Let the algorithm choose.
      val impurity = "gini"
      val maxDepth = 20
      val maxBins = 32
  
      val model = RandomForest.trainClassifier(pcaTrainingData, numClasses, categoricalFeaturesInfo,
          numTrees, featureSubsetStrategy, impurity, maxDepth, maxBins)
  
  
       error: type mismatch;
       found   : org.apache.spark.sql.DataFrame
       required: org.apache.spark.rdd.RDD[org.apache.spark.mllib.regression.LabeledPoint]
  </code></pre>
  
  <p>I tried the following two possible solutions but they didn't work:</p>
  
  <pre><code> scala&gt; val pcaTrainingData = trainingData.map(p =&gt; p.copy(features = pca.transform(p.features)))
   &lt;console&gt;:39: error: overloaded method value transform with alternatives:
     (dataset: org.apache.spark.sql.DataFrame)org.apache.spark.sql.DataFrame &lt;and&gt;
     (dataset: org.apache.spark.sql.DataFrame,paramMap: org.apache.spark.ml.param.ParamMap)org.apache.spark.sql.DataFrame &lt;and&gt;
     (dataset: org.apache.spark.sql.DataFrame,firstParamPair: org.apache.spark.ml.param.ParamPair[_],otherParamPairs: org.apache.spark.ml.param.ParamPair[_]*)org.apache.spark.sql.DataFrame
    cannot be applied to (org.apache.spark.mllib.linalg.Vector)
  </code></pre>
  
  <p>And:</p>
  
  <pre><code>     val labeled = pca
      .transform(trainingDf)
      .map(row =&gt; LabeledPoint(row.getDouble(0), row(4).asInstanceOf[Vector[Int]]))
  
       error: type mismatch;
       found   : scala.collection.immutable.Vector[Int]
       required: org.apache.spark.mllib.linalg.Vector
  </code></pre>
  
  <p>(I have imported org.apache.spark.mllib.linalg.Vectors in the above case)</p>
  
  <p>Any help?</p>