{"action":"created","issue":{"url":"https://api.github.com/repos/dmlc/xgboost/issues/3526","repository_url":"https://api.github.com/repos/dmlc/xgboost","labels_url":"https://api.github.com/repos/dmlc/xgboost/issues/3526/labels{/name}","comments_url":"https://api.github.com/repos/dmlc/xgboost/issues/3526/comments","events_url":"https://api.github.com/repos/dmlc/xgboost/issues/3526/events","html_url":"https://github.com/dmlc/xgboost/issues/3526","id":345487318,"node_id":"MDU6SXNzdWUzNDU0ODczMTg=","number":3526,"title":"[jvm-packages]Booster.predict gives different result than XGBoostModel.predict","user":{"login":"mliu60","id":39742459,"node_id":"MDQ6VXNlcjM5NzQyNDU5","avatar_url":"https://avatars0.githubusercontent.com/u/39742459?v=4","gravatar_id":"","url":"https://api.github.com/users/mliu60","html_url":"https://github.com/mliu60","followers_url":"https://api.github.com/users/mliu60/followers","following_url":"https://api.github.com/users/mliu60/following{/other_user}","gists_url":"https://api.github.com/users/mliu60/gists{/gist_id}","starred_url":"https://api.github.com/users/mliu60/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mliu60/subscriptions","organizations_url":"https://api.github.com/users/mliu60/orgs","repos_url":"https://api.github.com/users/mliu60/repos","events_url":"https://api.github.com/users/mliu60/events{/privacy}","received_events_url":"https://api.github.com/users/mliu60/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":14,"created_at":"2018-07-28T23:40:40Z","updated_at":"2018-07-29T00:37:44Z","closed_at":null,"author_association":"NONE","body":"Hi, I'm trying to train XGBoost on Spark and use the Booster object to label online traffic. However, I noticed that prediction given by XGBoostModel.predict differs from prediction given by Booster.predict. Is this the expected behavior or is it a bug? Here are the code snippet I used for training, evaluation and online labeling.\r\nCode snippet used for training:\r\n```        SparkConf conf = new SparkConf().setAppName(jobName);\r\n        conf.set(\"spark.serializer\", \"org.apache.spark.serializer.KryoSerializer\");\r\n        Class[] boosters = {ml.dmlc.xgboost4j.scala.Booster.class};\r\n        conf.registerKryoClasses(boosters);\r\n        SparkContext sc = new SparkContext(conf);\r\n        XGBoostModel model = train(sc);\r\n        evaluate1(sc, model);\r\n        evaluate2(sc, model);\r\n```\r\n```    private static XGBoostModel train(SparkContext sc) throws XGBoostError {\r\n        JavaRDD<LabeledPoint> trainRDD =\r\n            MLUtils.loadLibSVMFile(sc, inputTrainPath).toJavaRDD().\r\n                map((lp) -> new LabeledPoint(lp.label(), new DenseVector(lp.features().toArray())));\r\n\r\n        java.util.Map<String, Object> jparamMap = new java.util.HashMap<>();\r\n        jparamMap.put(\"eta\", eta);\r\n        jparamMap.put(\"max_depth\", maxDepth);\r\n        jparamMap.put(\"objective\", objective);\r\n\r\n        // use absolutely black magic to convert a Java Map to a Scala Immutable Map\r\n        Map<String, Object> paramMap = JavaConverters\r\n            .mapAsScalaMapConverter(jparamMap)\r\n            .asScala()\r\n            .toMap(Predef.conforms());\r\n\r\n        // Return the trained model.\r\n        return XGBoost.trainWithRDD(\r\n            JavaRDD.toRDD(trainRDD),\r\n            paramMap,\r\n            numRound,\r\n            numWorker,\r\n            null, // can specify a custom objective (type ObjectiveTrait)\r\n            null, // can specify a custom evaluation metric (type EvalTrait)\r\n            false,\r\n            missing);\r\n    }\r\n```\r\nPredict new data points using XGBoostModel:\r\n```    private static void evaluate1(SparkContext sc, XGBoostModel model) {\r\n        JavaRDD<DenseVector> testSet = MLUtils.loadLibSVMFile(sc, hdfsTestPath).toJavaRDD()\r\n            .map((lp) -> new DenseVector(lp.features().toArray()));\r\n        JavaRDD<float[]> predictions = model\r\n            .predict(JavaRDD.toRDD(testSet), missing).toJavaRDD();\r\n        predictions.map((pred) -> Arrays.toString(pred))\r\n            .saveAsTextFile(outputPredPath + \"1\");\r\n    }\r\n```\r\nPredict new data points using Booster:\r\n```    private static void evaluate2(SparkContext sc, XGBoostModel model) throws XGBoostError {\r\n        DMatrix testSet = new DMatrix(localTestPath);\r\n        JavaSparkContext.fromSparkContext(sc)\r\n            .parallelize(Arrays.asList(model.booster().predict(testSet, false, 0)))\r\n            .map((pred) -> Arrays.toString(pred))\r\n            .saveAsTextFile(outputPredPath + \"2\");\r\n    }\r\n```\r\nThanks for your help."},"comment":{"url":"https://api.github.com/repos/dmlc/xgboost/issues/comments/408643890","html_url":"https://github.com/dmlc/xgboost/issues/3526#issuecomment-408643890","issue_url":"https://api.github.com/repos/dmlc/xgboost/issues/3526","id":408643890,"node_id":"MDEyOklzc3VlQ29tbWVudDQwODY0Mzg5MA==","user":{"login":"hcho3","id":2532981,"node_id":"MDQ6VXNlcjI1MzI5ODE=","avatar_url":"https://avatars0.githubusercontent.com/u/2532981?v=4","gravatar_id":"","url":"https://api.github.com/users/hcho3","html_url":"https://github.com/hcho3","followers_url":"https://api.github.com/users/hcho3/followers","following_url":"https://api.github.com/users/hcho3/following{/other_user}","gists_url":"https://api.github.com/users/hcho3/gists{/gist_id}","starred_url":"https://api.github.com/users/hcho3/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hcho3/subscriptions","organizations_url":"https://api.github.com/users/hcho3/orgs","repos_url":"https://api.github.com/users/hcho3/repos","events_url":"https://api.github.com/users/hcho3/events{/privacy}","received_events_url":"https://api.github.com/users/hcho3/received_events","type":"User","site_admin":false},"created_at":"2018-07-29T00:37:44Z","updated_at":"2018-07-29T00:37:44Z","author_association":"CONTRIBUTOR","body":"@CodingCat In this case, updating dmlc-core submodule should be sufficient, since `indexing_mode` is being passed as a URL string to the dmlc-core LIBSVM parser."}}
{"actor":{"display_login":"hcho3"}}