{"action":"created","issue":{"url":"https://api.github.com/repos/zhreshold/mxnet-ssd/issues/30","repository_url":"https://api.github.com/repos/zhreshold/mxnet-ssd","labels_url":"https://api.github.com/repos/zhreshold/mxnet-ssd/issues/30/labels{/name}","comments_url":"https://api.github.com/repos/zhreshold/mxnet-ssd/issues/30/comments","events_url":"https://api.github.com/repos/zhreshold/mxnet-ssd/issues/30/events","html_url":"https://github.com/zhreshold/mxnet-ssd/issues/30","id":194501224,"number":30,"title":"Saving checkpoint failed !!","user":{"login":"Tyler-D","id":10151546,"avatar_url":"https://avatars.githubusercontent.com/u/10151546?v=3","gravatar_id":"","url":"https://api.github.com/users/Tyler-D","html_url":"https://github.com/Tyler-D","followers_url":"https://api.github.com/users/Tyler-D/followers","following_url":"https://api.github.com/users/Tyler-D/following{/other_user}","gists_url":"https://api.github.com/users/Tyler-D/gists{/gist_id}","starred_url":"https://api.github.com/users/Tyler-D/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Tyler-D/subscriptions","organizations_url":"https://api.github.com/users/Tyler-D/orgs","repos_url":"https://api.github.com/users/Tyler-D/repos","events_url":"https://api.github.com/users/Tyler-D/events{/privacy}","received_events_url":"https://api.github.com/users/Tyler-D/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-12-09T02:52:30Z","updated_at":"2016-12-09T16:53:50Z","closed_at":null,"body":"I used the framework to train the VGG16_reduced network on my own dataset. It trained well but I found it failed to save checkpoint. And the error traceback is as follows:\r\n```\r\nINFO:root:Saved checkpoint to \"/home/caffemaker/MXNet/mxnet/example/ssd/model/ssd_500-0001.params\"\r\n[10:26:02] /home/caffemaker/MXNet/mxnet/dmlc-core/include/dmlc/logging.h:235: [10:26:02] src/ndarray/ndarray.cc:231: Check failed: from.shape() == to->shape() operands shape mismatch\r\nTraceback (most recent call last):\r\n  File \"train_zface.py\", line 78, in <module>\r\n    args.lr_refactor_ratio, args.monitor, args.log_file)\r\n  File \"/home/caffemaker/MXNet/mxnet/example/ssd/train/train_net.py\", line 201, in train_net_on_others\r\n    monitor=monitor)\r\n  File \"/home/caffemaker/MXNet/mxnet/python/mxnet/module/base_module.py\", line 408, in fit\r\n    batch_end_callback=eval_batch_end_callback, epoch=epoch)\r\n  File \"/home/caffemaker/MXNet/mxnet/python/mxnet/module/base_module.py\", line 169, in score\r\n    self.forward(eval_batch, is_train=False)\r\n  File \"/home/caffemaker/MXNet/mxnet/python/mxnet/module/module.py\", line 388, in forward\r\n    self._exec_group.forward(data_batch, is_train)\r\n  File \"/home/caffemaker/MXNet/mxnet/python/mxnet/module/executor_group.py\", line 319, in forward\r\n    _load_label(data_batch, self.label_arrays, self.label_layouts)\r\n  File \"/home/caffemaker/MXNet/mxnet/python/mxnet/module/executor_group.py\", line 49, in _load_label\r\n    _load_general(batch.label, targets, major_axis)\r\n  File \"/home/caffemaker/MXNet/mxnet/python/mxnet/module/executor_group.py\", line 36, in _load_general\r\n    d_dst_copy.copyto(d_dst)\r\n  File \"/home/caffemaker/MXNet/mxnet/python/mxnet/ndarray.py\", line 533, in copyto\r\n    return _internal._copyto(self, out=other)\r\n  File \"/home/caffemaker/MXNet/mxnet/python/mxnet/ndarray.py\", line 1225, in unary_ndarray_function\r\n    c_array(ctypes.c_char_p, [c_str(str(i)) for i in kwargs.values()])))\r\n  File \"/home/caffemaker/MXNet/mxnet/python/mxnet/base.py\", line 77, in check_call\r\n    raise MXNetError(py_str(_LIB.MXGetLastError()))\r\nmxnet.base.MXNetError: [10:26:02] src/ndarray/ndarray.cc:231: Check failed: from.shape() == to->shape() operands shape mismatch\r\n```\r\nI changed the SSD_VGG16_Reduced architecture a little, including the from layers and the class numbers.\r\nAnd I tried the framework in this repository and the dmlc repository ---- Same error.\r\nDid I make some mistakes or is it a bug ? @zhreshold Thanks for help !"},"comment":{"url":"https://api.github.com/repos/zhreshold/mxnet-ssd/issues/comments/266063420","html_url":"https://github.com/zhreshold/mxnet-ssd/issues/30#issuecomment-266063420","issue_url":"https://api.github.com/repos/zhreshold/mxnet-ssd/issues/30","id":266063420,"user":{"login":"zhreshold","id":3307514,"avatar_url":"https://avatars.githubusercontent.com/u/3307514?v=3","gravatar_id":"","url":"https://api.github.com/users/zhreshold","html_url":"https://github.com/zhreshold","followers_url":"https://api.github.com/users/zhreshold/followers","following_url":"https://api.github.com/users/zhreshold/following{/other_user}","gists_url":"https://api.github.com/users/zhreshold/gists{/gist_id}","starred_url":"https://api.github.com/users/zhreshold/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zhreshold/subscriptions","organizations_url":"https://api.github.com/users/zhreshold/orgs","repos_url":"https://api.github.com/users/zhreshold/repos","events_url":"https://api.github.com/users/zhreshold/events{/privacy}","received_events_url":"https://api.github.com/users/zhreshold/received_events","type":"User","site_admin":false},"created_at":"2016-12-09T16:53:50Z","updated_at":"2016-12-09T16:53:50Z","body":"Saving checkpoint is done. The error occurs when validation is started. The label shape in validation set mismatch with the data. So I guess there's something wrong with your code.\r\n\r\nPost your modified train_net.py please."}}
{"actor":{"display_login":"zhreshold"}}