{"action":"created","issue":{"url":"https://api.github.com/repos/dmlc/xgboost/issues/1849","repository_url":"https://api.github.com/repos/dmlc/xgboost","labels_url":"https://api.github.com/repos/dmlc/xgboost/issues/1849/labels{/name}","comments_url":"https://api.github.com/repos/dmlc/xgboost/issues/1849/comments","events_url":"https://api.github.com/repos/dmlc/xgboost/issues/1849/events","html_url":"https://github.com/dmlc/xgboost/issues/1849","id":194283982,"number":1849,"title":"[jvm-package] discussion on evaluation of xgboost4j-spark","user":{"login":"fromradio","id":6701845,"avatar_url":"https://avatars.githubusercontent.com/u/6701845?v=3","gravatar_id":"","url":"https://api.github.com/users/fromradio","html_url":"https://github.com/fromradio","followers_url":"https://api.github.com/users/fromradio/followers","following_url":"https://api.github.com/users/fromradio/following{/other_user}","gists_url":"https://api.github.com/users/fromradio/gists{/gist_id}","starred_url":"https://api.github.com/users/fromradio/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fromradio/subscriptions","organizations_url":"https://api.github.com/users/fromradio/orgs","repos_url":"https://api.github.com/users/fromradio/repos","events_url":"https://api.github.com/users/fromradio/events{/privacy}","received_events_url":"https://api.github.com/users/fromradio/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":12,"created_at":"2016-12-08T09:19:57Z","updated_at":"2016-12-13T10:17:49Z","closed_at":null,"body":"Hi, all. Recently I am reading the source code of xgboost4j-spark. I found that the output of predict methods is kind of un-usable.\r\n\r\n```scala\r\n  def predict(testSet: RDD[MLDenseVector], missingValue: Float): RDD[Array[Array[Float]]] = {\r\n    val broadcastBooster = testSet.sparkContext.broadcast(_booster)\r\n    testSet.mapPartitions { testSamples =>\r\n      val sampleArray = testSamples.toList\r\n      val numRows = sampleArray.size\r\n      val numColumns = sampleArray.head.size\r\n      if (numRows == 0) {\r\n        Iterator()\r\n      } else {\r\n        val rabitEnv = Map(\"DMLC_TASK_ID\" -> TaskContext.getPartitionId().toString)\r\n        Rabit.init(rabitEnv.asJava)\r\n        // translate to required format\r\n        val flatSampleArray = new Array[Float](numRows * numColumns)\r\n        for (i <- flatSampleArray.indices) {\r\n          flatSampleArray(i) = sampleArray(i / numColumns).values(i % numColumns).toFloat\r\n        }\r\n        val dMatrix = new DMatrix(flatSampleArray, numRows, numColumns, missingValue)\r\n        Rabit.shutdown()\r\n        Iterator(broadcastBooster.value.predict(dMatrix))\r\n      }\r\n    }\r\n  }\r\n```\r\n\r\nThe output is a RDD of a matrix that each row represents the prediction result (I print the result it seems to be just one Float). OK, you now just get the each partitions' prediction result and you do not know the corresponding relation with the origin data. I don't know what I can do with such a result. I am wondering whether should we delete these methods or replace them with others. I suggest to offer two methods. One is to return input and prediction, the other is to return the label and prediction (enough for most evaluation).\r\n\r\nBTW I can offer my implementation."},"comment":{"url":"https://api.github.com/repos/dmlc/xgboost/issues/comments/266700269","html_url":"https://github.com/dmlc/xgboost/issues/1849#issuecomment-266700269","issue_url":"https://api.github.com/repos/dmlc/xgboost/issues/1849","id":266700269,"user":{"login":"fromradio","id":6701845,"avatar_url":"https://avatars.githubusercontent.com/u/6701845?v=3","gravatar_id":"","url":"https://api.github.com/users/fromradio","html_url":"https://github.com/fromradio","followers_url":"https://api.github.com/users/fromradio/followers","following_url":"https://api.github.com/users/fromradio/following{/other_user}","gists_url":"https://api.github.com/users/fromradio/gists{/gist_id}","starred_url":"https://api.github.com/users/fromradio/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fromradio/subscriptions","organizations_url":"https://api.github.com/users/fromradio/orgs","repos_url":"https://api.github.com/users/fromradio/repos","events_url":"https://api.github.com/users/fromradio/events{/privacy}","received_events_url":"https://api.github.com/users/fromradio/received_events","type":"User","site_admin":false},"created_at":"2016-12-13T10:17:49Z","updated_at":"2016-12-13T10:17:49Z","body":"@geoHeil Thanks a lot. It seems to be what I want! I will make a test on prediction speed with xgboost4j-spark I modified."}}
{"actor":{"display_login":"fromradio"}}