title: XGBoost-4j by DMLC on Spark-1.6.1
body: <p>I am trying to use the XGBoost implementation by DMLC on Spark-1.6.1. I am able to train my data with XGBoost but facing difficulties in prediction. I actually want to do prediction in the way it can be done in Apache Spark mllib libraries, that helps with calculation of training error,precision, recall, specificity etc.</p>
  
  <p>I am posting the codes below, also the error I am getting.
  I used this xgboost4j-spark-0.5-jar-with-dependencies.jar in spark-shell to start.</p>
  
  <pre><code>import org.apache.spark.mllib.regression.LabeledPoint
  import org.apache.spark.mllib.linalg.Vectors
  import org.apache.spark.mllib.util.MLUtils
  import org.apache.spark.SparkContext._
  import org.apache.spark.mllib.linalg.Vectors
  import org.apache.spark.mllib.regression.LabeledPoint
  import ml.dmlc.xgboost4j.scala.Booster
  import ml.dmlc.xgboost4j.scala.spark.XGBoost
  import ml.dmlc.xgboost4j.scala.DMatrix
  import ml.dmlc.xgboost4j.scala.{Booster, DMatrix}
  import ml.dmlc.xgboost4j.scala.spark.{DataUtils, XGBoost}
  import org.apache.spark.{SparkConf, SparkContext}
  
  
  
  
  //Load and parse the data file.
  val data = sc.textFile("file:///home/partha/credit_approval_2_attr.csv")
  val data1 = sc.textFile("file:///home/partha/credit_app_fea.csv")
  
  
  val parsedData = data.map { line =&gt;
      val parts = line.split(',').map(_.toDouble)
      LabeledPoint(parts(0), Vectors.dense(parts.tail))
  }.cache()
  
  val parsedData1 = data1.map { line =&gt;
      val parts = line.split(',').map(_.toDouble)
      Vectors.dense(parts)
  }
  
  
  
  //Tuning Parameters
  val paramMap = List(
        "eta" -&gt; 0.1f,  
        "max_depth" -&gt; 5,
        "num_class" -&gt; 2,
        "objective" -&gt; "multi:softmax" ,
        "colsample_bytree" -&gt; 0.8,
         "alpha" -&gt; 1,
         "subsample" -&gt; 0.5).toMap
  
    //Training the model  
  val numRound = 20
  val model = XGBoost.train(parsedData, paramMap, numRound, nWorkers = 1)
  val pred = model.predict(parsedData1)
  pred.collect()
  </code></pre>
  
  <p>Output from pred:</p>
  
  <pre><code>res0: Array[Array[Array[Float]]] = Array(Array(Array(0.0), Array(1.0), Array(1.0), Array(1.0), Array(0.0), Array(0.0), Array(1.0), Array(1.0), Array(0.0), Array(1.0), Array(0.0), Array(0.0), Array(0.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(0.0), Array(1.0), Array(1.0), Array(0.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(0.0), Array(1.0), Array(1.0), Array(1.0), Array(0.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(0.0), Array(0.0), Array(0.0), Array(0.0), Array(1.0), Array(0.0), Array(0.0), Array(0.0), Array(0.0), Array(0.0), Array(0.0), Array(1.0), Array(1.0), Array(1.0), Array(...
  </code></pre>
  
  <p>Now when I am using:</p>
  
  <pre><code>val labelAndPreds = parsedData.map { point =&gt;
            val prediction = model.predict(point.features)
            (point.label, prediction)
          }
  </code></pre>
  
  <p>Output:</p>
  
  <pre><code>&lt;console&gt;:66: error: overloaded method value predict with alternatives:
    (testSet: ml.dmlc.xgboost4j.scala.DMatrix)Array[Array[Float]] &lt;and&gt;
    (testSet: org.apache.spark.rdd.RDD[org.apache.spark.mllib.linalg.Vector])org.apache.spark.rdd.RDD[Array[Array[Float]]]
   cannot be applied to (org.apache.spark.mllib.linalg.Vector)
                    val prediction = model.predict(point.features)
                                       ^
  </code></pre>
  
  <p>And then tried this, since predict requires an RDD[Vector].</p>
  
  <pre><code>val labelAndPreds1 = parsedData.map { point =&gt;
            val prediction = model.predict(Vectors.dense(point.features))
            (point.label, prediction)
          }
  </code></pre>
  
  <p>The outcome was:</p>
  
  <pre><code>&lt;console&gt;:66: error: overloaded method value dense with alternatives:
    (values: Array[Double])org.apache.spark.mllib.linalg.Vector &lt;and&gt;
    (firstValue: Double,otherValues: Double*)org.apache.spark.mllib.linalg.Vector
   cannot be applied to (org.apache.spark.mllib.linalg.Vector)
                    val prediction = model.predict(Vectors.dense(point.features))
                                                           ^
  </code></pre>
  
  <p>Clearly its an issue of RDD type which I am trying to sort out, this is easy with GBT on spark (<a href="http://spark.apache.org/docs/latest/mllib-ensembles.html#gradient-boosted-trees-gbts" rel="nofollow">http://spark.apache.org/docs/latest/mllib-ensembles.html#gradient-boosted-trees-gbts</a>).</p>
  
  <p>Am I trying to do this the right way?</p>
  
  <p>Any help or suggestion would be awesome.</p>