{"action":"reopened","issue":{"url":"https://api.github.com/repos/dmlc/xgboost/issues/4037","repository_url":"https://api.github.com/repos/dmlc/xgboost","labels_url":"https://api.github.com/repos/dmlc/xgboost/issues/4037/labels{/name}","comments_url":"https://api.github.com/repos/dmlc/xgboost/issues/4037/comments","events_url":"https://api.github.com/repos/dmlc/xgboost/issues/4037/events","html_url":"https://github.com/dmlc/xgboost/issues/4037","id":395297319,"node_id":"MDU6SXNzdWUzOTUyOTczMTk=","number":4037,"title":"Segmentation fault while using external memory version","user":{"login":"kambstreat","id":1847554,"node_id":"MDQ6VXNlcjE4NDc1NTQ=","avatar_url":"https://avatars2.githubusercontent.com/u/1847554?v=4","gravatar_id":"","url":"https://api.github.com/users/kambstreat","html_url":"https://github.com/kambstreat","followers_url":"https://api.github.com/users/kambstreat/followers","following_url":"https://api.github.com/users/kambstreat/following{/other_user}","gists_url":"https://api.github.com/users/kambstreat/gists{/gist_id}","starred_url":"https://api.github.com/users/kambstreat/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kambstreat/subscriptions","organizations_url":"https://api.github.com/users/kambstreat/orgs","repos_url":"https://api.github.com/users/kambstreat/repos","events_url":"https://api.github.com/users/kambstreat/events{/privacy}","received_events_url":"https://api.github.com/users/kambstreat/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2019-01-02T16:10:43Z","updated_at":"2019-01-03T15:46:13Z","closed_at":null,"author_association":"NONE","body":"Hi...I am using the following code to run xgboost external memory version as the data I am using is huge. \r\n\r\n**Python code for xgboost external training.** \r\n\r\n```\r\nimport sys\r\nimport xgboost as xgb\r\nimport numpy as np\r\nimport multiprocessing\r\n\r\ntrain_file = 'train_data.txt'\r\nn_trees = 500\r\nnum_cpus = multiprocessing.cpu_count()\r\ntrain_file_with_cache = train_file + \"#dtrain.cache\"\r\n\r\ndtrain = xgb.DMatrix(train_file_with_cache)\r\nparam = {\r\n    'max_depth': 5,\r\n    'eta': 0.1,\r\n    'objective': 'binary:logistic',\r\n    'nthread': num_cpus\r\n}\r\nbst = xgb.train(param, dtrain, num_boost_round=n_trees)\r\n```\r\n\r\n**train_data.txt**\r\n\r\n```\r\n0 1:0.05 2:0.2886751346 3:0.1428571429 4:14.5185062529 5:18.9276504219 8:0.6666666667 9:0.8164965809 10:1000.0 11:10.0 12:0.0023387311 13:0.0002435188 14:0.0000963849 15:0.0063504397 16:0.00048034 17:0.0000850515 18:2.0 20:2.0 21:0.0235294118 22:11.0 23:3.0 24:2.0 25:1.0 26:11.0 27:3.0 28:2.0 29:1.0 31:1.0 34:1.0 37:1.0 40:20.0 41:0.5773502692 42:0.3333333333\r\n0 1:0.1315789474 2:0.2672612419 3:0.15 4:26.789135688 5:16.1891025786 8:1.0 9:1.0 10:2899.0 11:29.99 12:0.0183876189 13:0.000405548 14:0.0000554602 15:0.0101011466 16:0.0005706792 17:0.000093875 18:199.0 20:199.0 21:1.0310880829 22:1.0 23:1.0 24:1.0 25:1.0 26:1.0 27:1.0 28:1.0 29:1.0 31:1.0 34:1.0 37:1.0 40:7.0 41:0.5773502692 42:0.3333333333\r\n1 1:0.1111111111 2:0.4082482905 3:0.2222222222 4:23.1104210691 5:16.863139254 8:1.0 9:1.0 10:1700.0 11:17.0 12:0.0062726159 13:0.000491584 14:0.0003132148 15:0.0065093033 16:0.0005302828 17:0.0001788845 18:25.0 20:14.0 21:0.0370919881 22:4.0 23:1.0 24:1.0 25:1.0 26:4.0 27:1.0 28:1.0 29:1.0 31:1.0 34:1.0 37:1.0 40:2.0 41:1.0 42:1.0\r\n1 1:0.0178571429 2:0.2611164839 3:0.12 4:4.2300158408 5:16.6917285999 8:0.2 9:0.3333333333 10:1600.0 11:16.0 12:0.0042197019 13:0.000288361 14:0.0001064819 15:0.0076858998 16:0.0007074996 17:0.000212954 22:61.0 23:13.0 24:7.0 25:4.0 26:61.0 27:13.0 28:7.0 29:4.0 31:1.0 34:1.0 37:1.0 40:15.0 41:0.5773502692 42:0.3333333333\r\n0 1:0.0416666667 2:0.2581988897 3:0.0952380952 4:18.8293173733 5:12.3037244648 10:1995.0 11:20.95 12:0.0129559065 13:0.0008584607 14:0.0001920785 15:0.0102470342 16:0.0006167027 17:0.0002648885 18:57.0 20:57.0 21:0.2701421801 22:1.0 23:1.0 24:1.0 25:1.0 26:1.0 27:1.0 28:1.0 29:1.0 32:1.0 34:1.0 37:1.0 40:14.0 41:0.5 42:0.25\r\n1 1:0.0588235294 2:0.3396831102 3:0.1875 4:14.5185062529 5:20.3706356214 8:1.0 9:1.0 10:800.0 11:8.0 12:0.0027076417 13:0.0001515364 14:0.0000630457 15:0.0035674091 16:0.0002520206 17:0.0000883569 18:3.0 20:3.0 21:0.0357142857 22:6.0 23:2.0 24:1.0 25:1.0 26:6.0 27:2.0 28:1.0 29:1.0 31:1.0 34:1.0 37:1.0 40:6.0 41:0.5773502692 42:0.3333333333\r\n0 1:0.0625 2:0.3202563076 3:0.1428571429 4:25.767381348 5:19.023385007 8:0.3333333333 9:0.5773502692 10:1699.0 11:17.99 12:0.0167115449 13:0.0016991411 14:0.0004404174 15:0.007886134 16:0.0007152095 17:0.0002080609 18:1423.0 19:6.0 20:1277.0 21:3.3403755869 22:1.0 23:1.0 24:1.0 25:1.0 26:1.0 27:1.0 28:1.0 29:1.0 31:1.0 34:1.0 37:1.0 40:1.0 41:0.5773502692 42:0.3333333333\r\n0 1:0.0545454545 2:0.246182982 3:0.0869565217 4:12.2680659626 5:6.0386632079 10:2149.0 11:21.49 12:0.0100405638 13:0.0003642296 14:0.000050019 15:0.0073479124 16:0.0003586877 17:0.0000985799 18:245.0 20:200.0 21:0.374617737 22:1.0 23:1.0 24:1.0 25:1.0 26:1.0 27:1.0 28:1.0 29:1.0 31:1.0 34:1.0 37:1.0 40:3.0 41:0.5 42:0.25\r\n1 1:0.1272727273 2:0.298142397 3:0.16 4:43.0184411519 5:36.749580197 10:199.0 11:2.99 12:0.006621655 13:0.0012702954 14:0.0003402279 15:0.00645962 16:0.0006557264 17:0.0001699916 18:61.0 20:61.0 21:0.2489795918 22:6.0 23:2.0 24:1.0 25:1.0 26:6.0 27:2.0 28:1.0 29:1.0 31:1.0 34:1.0 37:1.0 40:7.0\r\n1 1:0.0263157895 2:0.15430335 3:0.0625 4:13.1159362977 5:15.5852962034 8:0.6 9:0.75 10:7900.0 11:79.0 12:0.0026556091 13:0.0002660176 14:0.0000962959 15:0.0075640712 16:0.0000997316 17:0.0000178409 18:16.0 20:4.0 21:0.0146386093 22:4.0 23:1.0 24:1.0 25:1.0 26:4.0 27:1.0 28:1.0 29:1.0 31:1.0 34:1.0 37:1.0 40:34.0 41:0.5 42:0.25\r\n```\r\n\r\nI ran the code with valgrind and I got the following error when I ran with 200k data instances but I did not get any error when I ran with 100k data instances. Please help me with the issue \r\n\r\n```\r\n==27020== Invalid read of size 4\r\n==27020==    at 0x1A1F2CF3: xgboost::tree::CQHistMaker<xgboost::tree::GradStats>::InitWorkSet(xgboost::DMatrix*, xgboost::RegTree const&, std::vector<unsigned int, std::allocator<unsigned int> >*) (in /home/username/python_debug_env/lib/python2.7/site-packages/xgboost-0.81-py2.7.egg/xgboost/lib/libxgboost.so)\r\n==27020==    by 0x1A1F697D: xgboost::tree::HistMaker<xgboost::tree::GradStats>::Update(xgboost::HostDeviceVector<xgboost::detail::GradientPairInternal<float> >*, xgboost::DMatrix*, std::vector<xgboost::RegTree*, std::allocator<xgboost::RegTree*> > const&) (in /home/username/python_debug_env/lib/python2.7/site-packages/xgboost-0.81-py2.7.egg/xgboost/lib/libxgboost.so)\r\n==27020==    by 0x1A25D75F: xgboost::gbm::GBTree::BoostNewTrees(xgboost::HostDeviceVector<xgboost::detail::GradientPairInternal<float> >*, xgboost::DMatrix*, int, std::vector<std::unique_ptr<xgboost::RegTree, std::default_delete<xgboost::RegTree> >, std::allocator<std::unique_ptr<xgboost::RegTree, std::default_delete<xgboost::RegTree> > > >*) (in /home/username/python_debug_env/lib/python2.7/site-packages/xgboost-0.81-py2.7.egg/xgboost/lib/libxgboost.so)\r\n==27020==    by 0x1A25EC1E: xgboost::gbm::GBTree::DoBoost(xgboost::DMatrix*, xgboost::HostDeviceVector<xgboost::detail::GradientPairInternal<float> >*, xgboost::ObjFunction*) (in /home/username/python_debug_env/lib/python2.7/site-packages/xgboost-0.81-py2.7.egg/xgboost/lib/libxgboost.so)\r\n==27020==    by 0x1A0E200D: xgboost::LearnerImpl::UpdateOneIter(int, xgboost::DMatrix*) (in /home/username/python_debug_env/lib/python2.7/site-packages/xgboost-0.81-py2.7.egg/xgboost/lib/libxgboost.so)\r\n==27020==    by 0x1A276664: XGBoosterUpdateOneIter (in /home/username/python_debug_env/lib/python2.7/site-packages/xgboost-0.81-py2.7.egg/xgboost/lib/libxgboost.so)\r\n==27020==    by 0x715EEB1: ffi_call_unix64 (unix64.S:76)\r\n==27020==    by 0x715E8D3: ffi_call (ffi64.c:525)\r\n==27020==    by 0x714D8F7: _call_function_pointer (callproc.c:836)\r\n==27020==    by 0x714E520: _ctypes_callproc (callproc.c:1179)\r\n==27020==    by 0x714710C: PyCFuncPtr_call (_ctypes.c:3965)\r\n==27020==    by 0x14B743: PyObject_Call (abstract.c:2546)\r\n==27020==  Address 0x1396a040 is 8 bytes after a block of size 344 alloc'd\r\n==27020==    at 0x4C2C21F: operator new(unsigned long) (vg_replace_malloc.c:334)\r\n==27020==    by 0x1A1F3384: xgboost::tree::CQHistMaker<xgboost::tree::GradStats>::InitWorkSet(xgboost::DMatrix*, xgboost::RegTree const&, std::vector<unsigned int, std::allocator<unsigned int> >*) (in /home/username/python_debug_env/lib/python2.7/site-packages/xgboost-0.81-py2.7.egg/xgboost/lib/libxgboost.so)\r\n==27020==    by 0x1A1F697D: xgboost::tree::HistMaker<xgboost::tree::GradStats>::Update(xgboost::HostDeviceVector<xgboost::detail::GradientPairInternal<float> >*, xgboost::DMatrix*, std::vector<xgboost::RegTree*, std::allocator<xgboost::RegTree*> > const&) (in /home/username/python_debug_env/lib/python2.7/site-packages/xgboost-0.81-py2.7.egg/xgboost/lib/libxgboost.so)\r\n==27020==    by 0x1A25D75F: xgboost::gbm::GBTree::BoostNewTrees(xgboost::HostDeviceVector<xgboost::detail::GradientPairInternal<float> >*, xgboost::DMatrix*, int, std::vector<std::unique_ptr<xgboost::RegTree, std::default_delete<xgboost::RegTree> >, std::allocator<std::unique_ptr<xgboost::RegTree, std::default_delete<xgboost::RegTree> > > >*) (in /home/username/python_debug_env/lib/python2.7/site-packages/xgboost-0.81-py2.7.egg/xgboost/lib/libxgboost.so)\r\n==27020==    by 0x1A25EC1E: xgboost::gbm::GBTree::DoBoost(xgboost::DMatrix*, xgboost::HostDeviceVector<xgboost::detail::GradientPairInternal<float> >*, xgboost::ObjFunction*) (in /home/username/python_debug_env/lib/python2.7/site-packages/xgboost-0.81-py2.7.egg/xgboost/lib/libxgboost.so)\r\n==27020==    by 0x1A0E200D: xgboost::LearnerImpl::UpdateOneIter(int, xgboost::DMatrix*) (in /home/username/python_debug_env/lib/python2.7/site-packages/xgboost-0.81-py2.7.egg/xgboost/lib/libxgboost.so)\r\n==27020==    by 0x1A276664: XGBoosterUpdateOneIter (in /home/username/python_debug_env/lib/python2.7/site-packages/xgboost-0.81-py2.7.egg/xgboost/lib/libxgboost.so)\r\n==27020==    by 0x715EEB1: ffi_call_unix64 (unix64.S:76)\r\n==27020==    by 0x715E8D3: ffi_call (ffi64.c:525)\r\n==27020==    by 0x714D8F7: _call_function_pointer (callproc.c:836)\r\n==27020==    by 0x714E520: _ctypes_callproc (callproc.c:1179)\r\n==27020==    by 0x714710C: PyCFuncPtr_call (_ctypes.c:3965)\r\n==27020== \r\n\r\n==6607== Invalid write of size 4\r\n==6607== at 0x1A1F2D21: xgboost::tree::CQHistMakerxgboost::tree::GradStats::InitWorkSet(xgboost::DMatrix*, xgboost::RegTree const&, std::vector<unsigned int, std::allocator >) (in /home/username/python_debug_env/lib/python2.7/site-packages/xgboost-0.81-py2.7.egg/xgboost/lib/libxgboost.so)\r\n==6607== by 0x1A1F697D: xgboost::tree::HistMakerxgboost::tree::GradStats::Update(xgboost::HostDeviceVector<xgboost::detail::GradientPairInternal >, xgboost::DMatrix*, std::vector<xgboost::RegTree*, std::allocatorxgboost::RegTree* > const&) (in /home/username/python_debug_env/lib/python2.7/site-packages/xgboost-0.81-py2.7.egg/xgboost/lib/libxgboost.so)\r\n==6607== by 0x1A25D75F: xgboost::gbm::GBTree::BoostNewTrees(xgboost::HostDeviceVector<xgboost::detail::GradientPairInternal >, xgboost::DMatrix, int, std::vector<std::unique_ptr<xgboost::RegTree, std::default_deletexgboost::RegTree >, std::allocator<std::unique_ptr<xgboost::RegTree, std::default_deletexgboost::RegTree > > >) (in /home/username/python_debug_env/lib/python2.7/site-packages/xgboost-0.81-py2.7.egg/xgboost/lib/libxgboost.so)\r\n==6607== by 0x1A25EC1E: xgboost::gbm::GBTree::DoBoost(xgboost::DMatrix, xgboost::HostDeviceVector<xgboost::detail::GradientPairInternal >, xgboost::ObjFunction) (in /home/username/python_debug_env/lib/python2.7/site-packages/xgboost-0.81-py2.7.egg/xgboost/lib/libxgboost.so)\r\n==6607== by 0x1A0E200D: xgboost::LearnerImpl::UpdateOneIter(int, xgboost::DMatrix*) (in /home/username/python_debug_env/lib/python2.7/site-packages/xgboost-0.81-py2.7.egg/xgboost/lib/libxgboost.so)\r\n==6607== by 0x1A276664: XGBoosterUpdateOneIter (in /home/username/python_debug_env/lib/python2.7/site-packages/xgboost-0.81-py2.7.egg/xgboost/lib/libxgboost.so)\r\n==6607== by 0x715EEB1: ffi_call_unix64 (unix64.S:76)\r\n==6607== by 0x715E8D3: ffi_call (ffi64.c:525)\r\n==6607== by 0x714D8F7: _call_function_pointer (callproc.c:836)\r\n==6607== by 0x714E520: _ctypes_callproc (callproc.c:1179)\r\n==6607== by 0x714710C: PyCFuncPtr_call (_ctypes.c:3965)\r\n==6607== by 0x14B743: PyObject_Call (abstract.c:2546)\r\n==6607== Address 0x5dca6f4 is 12 bytes after a block of size 344 alloc’d\r\n==6607== at 0x4C2C21F: operator new(unsigned long) (vg_replace_malloc.c:334)\r\n==6607== by 0x1A1F3384: xgboost::tree::CQHistMakerxgboost::tree::GradStats::InitWorkSet(xgboost::DMatrix*, xgboost::RegTree const&, std::vector<unsigned int, std::allocator >) (in /home/username/python_debug_env/lib/python2.7/site-packages/xgboost-0.81-py2.7.egg/xgboost/lib/libxgboost.so)\r\n==6607== by 0x1A1F697D: xgboost::tree::HistMakerxgboost::tree::GradStats::Update(xgboost::HostDeviceVector<xgboost::detail::GradientPairInternal >, xgboost::DMatrix*, std::vector<xgboost::RegTree*, std::allocatorxgboost::RegTree* > const&) (in /home/username/python_debug_env/lib/python2.7/site-packages/xgboost-0.81-py2.7.egg/xgboost/lib/libxgboost.so)\r\n==6607== by 0x1A25D75F: xgboost::gbm::GBTree::BoostNewTrees(xgboost::HostDeviceVector<xgboost::detail::GradientPairInternal >, xgboost::DMatrix, int, std::vector<std::unique_ptr<xgboost::RegTree, std::default_deletexgboost::RegTree >, std::allocator<std::unique_ptr<xgboost::RegTree, std::default_deletexgboost::RegTree > > >) (in /home/username/python_debug_env/lib/python2.7/site-packages/xgboost-0.81-py2.7.egg/xgboost/lib/libxgboost.so)\r\n==6607== by 0x1A25EC1E: xgboost::gbm::GBTree::DoBoost(xgboost::DMatrix, xgboost::HostDeviceVector<xgboost::detail::GradientPairInternal >, xgboost::ObjFunction) (in /home/username/python_debug_env/lib/python2.7/site-packages/xgboost-0.81-py2.7.egg/xgboost/lib/libxgboost.so)\r\n==6607== by 0x1A0E200D: xgboost::LearnerImpl::UpdateOneIter(int, xgboost::DMatrix*) (in /home/username/python_debug_env/lib/python2.7/site-packages/xgboost-0.81-py2.7.egg/xgboost/lib/libxgboost.so)\r\n==6607== by 0x1A276664: XGBoosterUpdateOneIter (in /home/username/python_debug_env/lib/python2.7/site-packages/xgboost-0.81-py2.7.egg/xgboost/lib/libxgboost.so)\r\n==6607== by 0x715EEB1: ffi_call_unix64 (unix64.S:76)\r\n==6607== by 0x715E8D3: ffi_call (ffi64.c:525)\r\n==6607== by 0x714D8F7: _call_function_pointer (callproc.c:836)\r\n==6607== by 0x714E520: _ctypes_callproc (callproc.c:1179)\r\n==6607== by 0x714710C: PyCFuncPtr_call (_ctypes.c:3965)\r\n==6607==\r\n==6607== Invalid read of size 8\r\n==6607== at 0x24D762: visit_decref (gcmodule.c:360)\r\n==6607== by 0x18797A: dict_traverse (dictobject.c:2114)\r\n==6607== by 0x24D853: subtract_refs (gcmodule.c:385)\r\n==6607== by 0x24E8B5: collect (gcmodule.c:925)\r\n==6607== by 0x24F639: PyGC_Collect (gcmodule.c:1440)\r\n==6607== by 0x233A07: Py_Finalize (pythonrun.c:448)\r\n==6607== by 0x140308: Py_Main (main.c:665)\r\n==6607== by 0x13EDFF: main (python.c:23)\r\n==6607== Address 0x3f8000000051f238 is not stack’d, malloc’d or (recently) free’d\r\n==6607==\r\n==6607==\r\n==6607== Process terminating with default action of signal 11 (SIGSEGV)\r\n==6607== General Protection Fault\r\n==6607== at 0x24D762: visit_decref (gcmodule.c:360)\r\n==6607== by 0x18797A: dict_traverse (dictobject.c:2114)\r\n==6607== by 0x24D853: subtract_refs (gcmodule.c:385)\r\n==6607== by 0x24E8B5: collect (gcmodule.c:925)\r\n==6607== by 0x24F639: PyGC_Collect (gcmodule.c:1440)\r\n==6607== by 0x233A07: Py_Finalize (pythonrun.c:448)\r\n==6607== by 0x140308: Py_Main (main.c:665)\r\n==6607== by 0x13EDFF: main (python.c:23)\r\n==6607==\r\n==6607== HEAP SUMMARY:\r\n==6607== in use at exit: 214,472,668 bytes in 68,975 blocks\r\n==6607== total heap usage: 197,280 allocs, 128,305 frees, 967,454,846 bytes allocated\r\n==6607==\r\n==6607== LEAK SUMMARY:\r\n==6607== definitely lost: 104 bytes in 2 blocks\r\n==6607== indirectly lost: 0 bytes in 0 blocks\r\n==6607== possibly lost: 3,167,370 bytes in 18,705 blocks\r\n==6607== still reachable: 211,305,162 bytes in 50,267 blocks\r\n==6607== of which reachable via heuristic:\r\n==6607== newarray : 560 bytes in 35 blocks\r\n==6607== suppressed: 32 bytes in 1 blocks\r\n==6607== Rerun with --leak-check=full to see details of leaked memory\r\n==6607==\r\n==6607== For counts of detected and suppressed errors, rerun with: -v\r\n==6607== ERROR SUMMARY: 69 errors from 5 contexts (suppressed: 0 from 0```"}}
{"actor":{"display_login":"trivialfis"}}