{"action":"closed","issue":{"url":"https://api.github.com/repos/msracver/Relation-Networks-for-Object-Detection/issues/13","repository_url":"https://api.github.com/repos/msracver/Relation-Networks-for-Object-Detection","labels_url":"https://api.github.com/repos/msracver/Relation-Networks-for-Object-Detection/issues/13/labels{/name}","comments_url":"https://api.github.com/repos/msracver/Relation-Networks-for-Object-Detection/issues/13/comments","events_url":"https://api.github.com/repos/msracver/Relation-Networks-for-Object-Detection/issues/13/events","html_url":"https://github.com/msracver/Relation-Networks-for-Object-Detection/issues/13","id":335357589,"node_id":"MDU6SXNzdWUzMzUzNTc1ODk=","number":13,"title":"mxnet.base.MXNetError: NaiveEngine only support synchronize Push so far","user":{"login":"Kongsea","id":4168984,"node_id":"MDQ6VXNlcjQxNjg5ODQ=","avatar_url":"https://avatars2.githubusercontent.com/u/4168984?v=4","gravatar_id":"","url":"https://api.github.com/users/Kongsea","html_url":"https://github.com/Kongsea","followers_url":"https://api.github.com/users/Kongsea/followers","following_url":"https://api.github.com/users/Kongsea/following{/other_user}","gists_url":"https://api.github.com/users/Kongsea/gists{/gist_id}","starred_url":"https://api.github.com/users/Kongsea/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Kongsea/subscriptions","organizations_url":"https://api.github.com/users/Kongsea/orgs","repos_url":"https://api.github.com/users/Kongsea/repos","events_url":"https://api.github.com/users/Kongsea/events{/privacy}","received_events_url":"https://api.github.com/users/Kongsea/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-06-25T10:58:38Z","updated_at":"2018-06-26T03:39:17Z","closed_at":"2018-06-26T03:39:17Z","author_association":"NONE","body":"After training for several epoches, it raised the following error:\r\n\r\n```\r\nEpoch[7] Batch [2280]\tSpeed: 3.99 samples/sec\tTrain-RPNAcc=0.995009,\tRPNLogLoss=0.016979,\tRPNL1Loss=0.034412,\tRCNNAcc=0.797916,\tRCNNLogLoss=0.433694,\tRCNNL1Loss=0.423676,\t\r\nEpoch[7] Batch [2300]\tSpeed: 3.96 samples/sec\tTrain-RPNAcc=0.995029,\tRPNLogLoss=0.016926,\tRPNL1Loss=0.034388,\tRCNNAcc=0.797866,\tRCNNLogLoss=0.433422,\tRCNNL1Loss=0.423580,\t\r\nEpoch[7] Batch [2320]\tSpeed: 3.69 samples/sec\tTrain-RPNAcc=0.995011,\tRPNLogLoss=0.017027,\tRPNL1Loss=0.034292,\tRCNNAcc=0.798346,\tRCNNLogLoss=0.432565,\tRCNNL1Loss=0.422573,\r\n[17:51:53] /home/fallingdust/workspace/mxnet/dmlc-core/include/dmlc/./logging.h:308: [17:51:53] src/engine/naive_engine.cc:168: Check failed: this->req_completed_ NaiveEngine only support synchronize Push so far\r\n\r\nStack trace returned 10 entries:\r\n[bt] (0) /usr/local/lib/python2.7/dist-packages/mxnet-1.0.0-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet6engine11NaiveEngine9PushAsyncESt8functionIFvNS_10RunContextENS0_18CallbackOnCompleteEEENS_7ContextERKSt6vectorIPNS0_3VarESaISA_EESE_NS_10FnPropertyEiPKc+0x3b3) [0x7f921cb635a3]\r\n[bt] (1) /usr/local/lib/python2.7/dist-packages/mxnet-1.0.0-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet6engine11NaiveEngine4PushEPNS0_3OprENS_7ContextEib+0x8f) [0x7f921cb644af]\r\n[bt] (2) /usr/local/lib/python2.7/dist-packages/mxnet-1.0.0-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet4exec13GraphExecutor6RunOpsEbmm+0x724) [0x7f921cc08e84]\r\n[bt] (3) /usr/local/lib/python2.7/dist-packages/mxnet-1.0.0-py2.7.egg/mxnet/libmxnet.so(MXExecutorForward+0x11) [0x7f921cb9ab81]\r\n[bt] (4) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call_unix64+0x4c) [0x7f9230c02e40]\r\n[bt] (5) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call+0x2eb) [0x7f9230c028ab]\r\n[bt] (6) /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so(_ctypes_callproc+0x48f) [0x7f9230e123df]\r\n[bt] (7) /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so(+0x11d82) [0x7f9230e16d82]\r\n[bt] (8) python(PyObject_Call+0x43) [0x4b0c93]\r\n[bt] (9) python(PyEval_EvalFrameEx+0x602f) [0x4c9f9f]\r\n\r\nTraceback (most recent call last):\r\n  File \"experiments/relation_rcnn/rcnn_end2end_train_test.py\", line 21, in <module>\r\n    train_end2end.main()\r\n  File \"experiments/relation_rcnn/../../relation_rcnn/train_end2end.py\", line 193, in main\r\n    config.TRAIN.begin_epoch, config.TRAIN.end_epoch, config.TRAIN.lr, config.TRAIN.lr_step)\r\n  File \"experiments/relation_rcnn/../../relation_rcnn/train_end2end.py\", line 186, in train_net\r\n    arg_params=arg_params, aux_params=aux_params, begin_epoch=begin_epoch, num_epoch=end_epoch)\r\n  File \"experiments/relation_rcnn/../../relation_rcnn/core/module.py\", line 999, in fit\r\n    self.forward_backward(data_batch)\r\n  File \"/usr/local/lib/python2.7/dist-packages/mxnet-1.0.0-py2.7.egg/mxnet/module/base_module.py\", line 191, in forward_backward\r\n    self.forward(data_batch, is_train=True)\r\n  File \"experiments/relation_rcnn/../../relation_rcnn/core/module.py\", line 1074, in forward\r\n    self._curr_module.forward(data_batch, is_train=is_train)\r\n  File \"experiments/relation_rcnn/../../relation_rcnn/core/module.py\", line 554, in forward\r\n    self._exec_group.forward(data_batch, is_train)\r\n  File \"experiments/relation_rcnn/../../relation_rcnn/core/DataParallelExecutorGroup.py\", line 360, in forward\r\n    exec_.forward(is_train=is_train)\r\n  File \"/usr/local/lib/python2.7/dist-packages/mxnet-1.0.0-py2.7.egg/mxnet/executor.py\", line 150, in forward\r\n    ctypes.c_int(int(is_train))))\r\n  File \"/usr/local/lib/python2.7/dist-packages/mxnet-1.0.0-py2.7.egg/mxnet/base.py\", line 146, in check_call\r\n    raise MXNetError(py_str(_LIB.MXGetLastError()))\r\nmxnet.base.MXNetError: [17:51:53] src/engine/naive_engine.cc:168: Check failed: this->req_completed_ NaiveEngine only support synchronize Push so far\r\n\r\nStack trace returned 10 entries:\r\n[bt] (0) /usr/local/lib/python2.7/dist-packages/mxnet-1.0.0-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet6engine11NaiveEngine9PushAsyncESt8functionIFvNS_10RunContextENS0_18CallbackOnCompleteEEENS_7ContextERKSt6vectorIPNS0_3VarESaISA_EESE_NS_10FnPropertyEiPKc+0x3b3) [0x7f921cb635a3]\r\n[bt] (1) /usr/local/lib/python2.7/dist-packages/mxnet-1.0.0-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet6engine11NaiveEngine4PushEPNS0_3OprENS_7ContextEib+0x8f) [0x7f921cb644af]\r\n[bt] (2) /usr/local/lib/python2.7/dist-packages/mxnet-1.0.0-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet4exec13GraphExecutor6RunOpsEbmm+0x724) [0x7f921cc08e84]\r\n[bt] (3) /usr/local/lib/python2.7/dist-packages/mxnet-1.0.0-py2.7.egg/mxnet/libmxnet.so(MXExecutorForward+0x11) [0x7f921cb9ab81]\r\n[bt] (4) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call_unix64+0x4c) [0x7f9230c02e40]\r\n[bt] (5) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call+0x2eb) [0x7f9230c028ab]\r\n[bt] (6) /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so(_ctypes_callproc+0x48f) [0x7f9230e123df]\r\n[bt] (7) /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so(+0x11d82) [0x7f9230e16d82]\r\n[bt] (8) python(PyObject_Call+0x43) [0x4b0c93]\r\n[bt] (9) python(PyEval_EvalFrameEx+0x602f) [0x4c9f9f]\r\n\r\n[17:51:53] src/engine/naive_engine.cc:55: Engine shutdown\r\n```\r\n\r\nI trained the network on just one GPU using `CUDA_VISIBLE_DEVICES=0` although I have two GPUs.\r\nPlease give me some advice or help to fix it. Thank you."}}
{"actor":{"display_login":"Kongsea"}}