{"action":"created","issue":{"url":"https://api.github.com/repos/dmlc/xgboost/issues/1176","repository_url":"https://api.github.com/repos/dmlc/xgboost","labels_url":"https://api.github.com/repos/dmlc/xgboost/issues/1176/labels{/name}","comments_url":"https://api.github.com/repos/dmlc/xgboost/issues/1176/comments","events_url":"https://api.github.com/repos/dmlc/xgboost/issues/1176/events","html_url":"https://github.com/dmlc/xgboost/issues/1176","id":153204752,"number":1176,"title":"[jvm-package]why the operation efficiency of xgboost4j-spark is so slow?","user":{"login":"youngjt","id":1442847,"avatar_url":"https://avatars.githubusercontent.com/u/1442847?v=3","gravatar_id":"","url":"https://api.github.com/users/youngjt","html_url":"https://github.com/youngjt","followers_url":"https://api.github.com/users/youngjt/followers","following_url":"https://api.github.com/users/youngjt/following{/other_user}","gists_url":"https://api.github.com/users/youngjt/gists{/gist_id}","starred_url":"https://api.github.com/users/youngjt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/youngjt/subscriptions","organizations_url":"https://api.github.com/users/youngjt/orgs","repos_url":"https://api.github.com/users/youngjt/repos","events_url":"https://api.github.com/users/youngjt/events{/privacy}","received_events_url":"https://api.github.com/users/youngjt/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":9,"created_at":"2016-05-05T10:19:46Z","updated_at":"2016-05-14T01:51:01Z","closed_at":null,"body":"We have found that the operation efficiency of xgboost4j-spark is particularly slow.See the following two experiments:\r\n\r\nThe size of the data set we used is: the number of samples is about 10160000, the feature dimension of each sample is 400.\r\n\r\n1:Stand-alone experiment:\r\n(1)CPU:Two 8 core CPU, 32  Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz,\r\n(2)Memory:64G\r\n**(3)training time is 41 minutes or so**\r\n\r\n2:Cluster experiment：\r\n(1)Specific settings for the spark-submit:：\r\nTRAIN_PATH=/tmp/xxxx/tmp_60\r\nTEST_PATH=/tmp/xxxxx/tmp_8\r\nOUTPUT_PATH=/tmp/xxxxxx\r\nnumRound=100\r\nnWorks=32\r\nJARPATH=/home/xxxxx/xgboost/jvm-packages/xgboost4j-example/target\r\nhadoop fs -rm -r $OUTPUT_PATH\r\nspark-submit \\ \r\n--class ml.dmlc.xgboost4j.scala.example.spark.DistTrainWithSpark\r\n--master yarn-client \\\r\n--num-executors 16 \\\r\n--driver-memory 4g \\\r\n--executor-cores 2 \\\r\n--executor-memory 20g \\\r\n--queue xxxxx \\\r\n$JARPATH/xgboost4j-example-0.5-jar-with-dependencies.jar $numRound $nWorks $TRAIN_PATH $TEST_PATH $OUTPUT_PATH\r\n**(2)training time is 92 minutes or so**\r\n\r\nWe found the \"exception\" information in the xgboost4j-spark distributed output log,as follows(Detailed information,see Annex1.txt)：\r\n**WARN Dispatcher: Message RemoteProcessDisconnected(BJHC-HADOOP-Hecate-10014.hadoop.jd.local:3493) dropped.\r\njava.lang.IllegalStateException: RpcEnv already stopped.\r\n\tat org.apache.spark.rpc.netty.Dispatcher.postMessage(Dispatcher.scala:159)\r\n\tat org.apache.spark.rpc.netty.Dispatcher.postToAll(Dispatcher.scala:109)\r\n\tat org.apache.spark.rpc.netty.NettyRpcHandler.connectionTerminated(NettyRpcEnv.scala:630)\r\n\tat **org.apache.spark.network.server.TransportRequestHandler.channelUnregistered(TransportRequestHandler.java:94)\r\n\tat org.apache.spark.network.server.TransportChannelHandler.channelUnregistered(TransportChannelHandler.java:89)\r\n\r\nWhen our training data is changed to a biger set which has about 20000000 samples and feature dimension of each sample is 400, we get no results after a very long time(Detailed information,see Annex2.txt).\r\n\r\nBy monitoring the program execution,we have found the execution of the program stays in the 104th row of ml.dmlc.xgboost4j.scala.spark , and related code is as follows：\r\n**102 val sparkJobThread = new Thread() {\r\n103     override def run() {\r\n104         boosters.foreachPartition(() => _)\r\n105     }\r\n106 }**\r\n\r\nFinally,my question is:\r\n**(1)Is this normal?\r\n(2)How to make the xgboost4j-spark run faster?**\r\n\r\n\r\n[Annex1.txt](https://github.com/dmlc/xgboost/files/250249/Annex1.txt)\r\n[Annex2.txt](https://github.com/dmlc/xgboost/files/250248/Annex2.txt)\r\n"},"comment":{"url":"https://api.github.com/repos/dmlc/xgboost/issues/comments/219194150","html_url":"https://github.com/dmlc/xgboost/issues/1176#issuecomment-219194150","issue_url":"https://api.github.com/repos/dmlc/xgboost/issues/1176","id":219194150,"user":{"login":"youngjt","id":1442847,"avatar_url":"https://avatars.githubusercontent.com/u/1442847?v=3","gravatar_id":"","url":"https://api.github.com/users/youngjt","html_url":"https://github.com/youngjt","followers_url":"https://api.github.com/users/youngjt/followers","following_url":"https://api.github.com/users/youngjt/following{/other_user}","gists_url":"https://api.github.com/users/youngjt/gists{/gist_id}","starred_url":"https://api.github.com/users/youngjt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/youngjt/subscriptions","organizations_url":"https://api.github.com/users/youngjt/orgs","repos_url":"https://api.github.com/users/youngjt/repos","events_url":"https://api.github.com/users/youngjt/events{/privacy}","received_events_url":"https://api.github.com/users/youngjt/received_events","type":"User","site_admin":false},"created_at":"2016-05-14T01:51:01Z","updated_at":"2016-05-14T01:51:01Z","body":"we have checked the parameters and the output log of track.py carefully, this problem is indeed."}}