{"action":"created","issue":{"url":"https://api.github.com/repos/apache/incubator-mxnet/issues/9939","repository_url":"https://api.github.com/repos/apache/incubator-mxnet","labels_url":"https://api.github.com/repos/apache/incubator-mxnet/issues/9939/labels{/name}","comments_url":"https://api.github.com/repos/apache/incubator-mxnet/issues/9939/comments","events_url":"https://api.github.com/repos/apache/incubator-mxnet/issues/9939/events","html_url":"https://github.com/apache/incubator-mxnet/pull/9939","id":301289068,"number":9939,"title":"add multi proposal operator (cpu version) and fix the bug in proposal op (gpu version)","user":{"login":"wkcn","id":3415417,"avatar_url":"https://avatars1.githubusercontent.com/u/3415417?v=4","gravatar_id":"","url":"https://api.github.com/users/wkcn","html_url":"https://github.com/wkcn","followers_url":"https://api.github.com/users/wkcn/followers","following_url":"https://api.github.com/users/wkcn/following{/other_user}","gists_url":"https://api.github.com/users/wkcn/gists{/gist_id}","starred_url":"https://api.github.com/users/wkcn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wkcn/subscriptions","organizations_url":"https://api.github.com/users/wkcn/orgs","repos_url":"https://api.github.com/users/wkcn/repos","events_url":"https://api.github.com/users/wkcn/events{/privacy}","received_events_url":"https://api.github.com/users/wkcn/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2018-03-01T06:10:17Z","updated_at":"2018-03-02T02:13:40Z","closed_at":null,"author_association":"NONE","pull_request":{"url":"https://api.github.com/repos/apache/incubator-mxnet/pulls/9939","html_url":"https://github.com/apache/incubator-mxnet/pull/9939","diff_url":"https://github.com/apache/incubator-mxnet/pull/9939.diff","patch_url":"https://github.com/apache/incubator-mxnet/pull/9939.patch"},"body":"## Description ##\r\nThe multi_proposal operator (`mxnet.sym.contrib.MultiProposal`, CPU version) is not implemented before.\r\nI wrote the code about it.\r\n\r\nAnd I found there was a bug in *proposal.cu* and *multi_proposal.cu*.\r\nThe `batch_size` of the output of *Proposal* and *MultiProposal* are both `param_.rpn_post_nms_top_n`.\r\nWhen `count_anchors < param_.rpn_post_nms_top_n`, the variable `rpn_post_nms_top_n` will be `count_anchors`, which is less than `param_.rpn_post_nms_top_n`.\r\nhttps://github.com/apache/incubator-mxnet/blob/master/src/operator/contrib/proposal.cu#L438\r\n\r\nIt will cause the problem that the output whose index is larger than `rpn_post_nms_top_n` will be *not assigned*.\r\n\r\n## Checklist ##\r\n### Essentials ###\r\n- [x] Passed code style checking (`make lint`)\r\n- [x] Changes are complete (i.e. I finished coding on this PR)\r\n- [x] All changes have test coverage:\r\n- Unit tests are added for small changes to verify correctness (e.g. adding a new operator)\r\n- Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)\r\n- Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)\r\n- [x] Code is well-documented: \r\n- For user-facing API changes, API doc string has been updated. \r\n- For new C++ functions in header files, their functionalities and arguments are documented. \r\n- For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable\r\n- [x] To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change\r\n\r\n### Changes ###\r\n- [x] Multi Proposal Operator is supported on the CPU context.\r\n- [x] Add unit-test for Multi Proposal Operator\r\n- [x] Fix `PrepareOutput` bug in proposal op (GPU) and multi_proposal op(GPU)\r\n\r\n## Comments ##\r\nHere is the testing code.\r\n```python\r\nimport mxnet as mx\r\nfrom mxnet import nd\r\nimport numpy as np\r\n\r\nfeature_stride = 16 \r\nscales = (8, 16, 32)\r\nratios = (0.5, 1, 2)\r\nrpn_pre_nms_top_n = 12000 \r\nrpn_post_nms_top_n = 2000\r\nthreshold = 0.7\r\nrpn_min_size = 16\r\n\r\nbatch_size = 20\r\nfeat_len = 14\r\nH, W = feat_len, feat_len\r\nnum_anchors = len(scales) * len(ratios)\r\ncount_anchors = feat_len * feat_len * num_anchors\r\n\r\n'''\r\n    cls_prob: (batch_size, 2 * num_anchors, H, W)\r\n    bbox_pred: (batch_size, 4 * num_anchors, H, W) \r\n    im_info: (batch_size, 3)\r\n'''\r\n\r\ncls_prob = nd.empty((batch_size, 2 * num_anchors, H, W), dtype = np.float32) \r\nbbox_pred = nd.empty((batch_size, 4 * num_anchors, H, W), dtype = np.float32) \r\nim_info = nd.empty((batch_size, 3), dtype = np.float32) \r\n\r\ncls_prob = nd.array(np.random.random(cls_prob.shape))\r\nbbox_pred = nd.array(np.random.random(bbox_pred.shape))\r\n\r\nfor i in range(batch_size):\r\n    im_size = np.random.randint(100, feat_len * feature_stride, size = (2,))\r\n    im_scale = np.random.randint(70, 100) / 100.0\r\n    im_info[i, :] = [im_size[0], im_size[1], im_scale]\r\n\r\ndef get_sub(arr, i):\r\n    new_shape = list(arr.shape)\r\n    new_shape[0] = 1\r\n    res = arr[i].reshape(new_shape)\r\n    return res\r\n\r\ndef test(rpn_pre_nms_top_n, rpn_post_nms_top_n):\r\n\r\n    single_proposal = []\r\n    single_score = []\r\n    for i in range(batch_size):\r\n        rois, score = mx.nd.contrib.Proposal(\r\n                cls_score = get_sub(cls_prob, i),\r\n                bbox_pred = get_sub(bbox_pred, i),\r\n                im_info = get_sub(im_info, i),\r\n                feature_stride = feature_stride,\r\n                scales = scales,\r\n                ratios = ratios,\r\n                rpn_pre_nms_top_n = rpn_pre_nms_top_n,\r\n                rpn_post_nms_top_n = rpn_post_nms_top_n,\r\n                threshold = threshold,\r\n                rpn_min_size = rpn_min_size, output_score = True)\r\n        single_proposal.append(rois)\r\n        single_score.append(score)\r\n\r\n    multi_proposal, multi_score = mx.nd.contrib.MultiProposal(\r\n            cls_score = cls_prob,\r\n            bbox_pred = bbox_pred,\r\n            im_info = im_info,\r\n            feature_stride = feature_stride,\r\n            scales = scales,\r\n            ratios = ratios,\r\n            rpn_pre_nms_top_n = rpn_pre_nms_top_n,\r\n            rpn_post_nms_top_n = rpn_post_nms_top_n,\r\n            threshold = threshold,\r\n            rpn_min_size = rpn_min_size, output_score = True)\r\n\r\n    single_proposal = nd.stack(*single_proposal).reshape(multi_proposal.shape)\r\n    single_score = nd.stack(*single_score).reshape(multi_score.shape)\r\n\r\n    single_proposal_np = single_proposal.asnumpy()\r\n    multi_proposal_np = multi_proposal.asnumpy()\r\n\r\n    single_score_np = single_score.asnumpy()\r\n    multi_score_np = multi_score.asnumpy()\r\n    print (multi_proposal_np.shape)\r\n\r\n    # test rois x1,y1,x2,y2\r\n    assert np.allclose(single_proposal_np[:, 1:], multi_proposal_np[:, 1:])\r\n    # test rois batch_idx \r\n    for i in range(batch_size):\r\n        start = i * rpn_post_nms_top_n\r\n        end = start + rpn_post_nms_top_n\r\n        assert (multi_proposal_np[start:end, 0] == i).all()\r\n\r\n    # test score\r\n    assert np.allclose(single_score_np, multi_score_np)\r\n\r\ntest(rpn_pre_nms_top_n, rpn_post_nms_top_n)\r\ntest(rpn_pre_nms_top_n, 1500)\r\ntest(1000, 500)\r\nprint (\"test ok\")\r\n\r\n```"},"comment":{"url":"https://api.github.com/repos/apache/incubator-mxnet/issues/comments/369797499","html_url":"https://github.com/apache/incubator-mxnet/pull/9939#issuecomment-369797499","issue_url":"https://api.github.com/repos/apache/incubator-mxnet/issues/9939","id":369797499,"user":{"login":"wkcn","id":3415417,"avatar_url":"https://avatars1.githubusercontent.com/u/3415417?v=4","gravatar_id":"","url":"https://api.github.com/users/wkcn","html_url":"https://github.com/wkcn","followers_url":"https://api.github.com/users/wkcn/followers","following_url":"https://api.github.com/users/wkcn/following{/other_user}","gists_url":"https://api.github.com/users/wkcn/gists{/gist_id}","starred_url":"https://api.github.com/users/wkcn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wkcn/subscriptions","organizations_url":"https://api.github.com/users/wkcn/orgs","repos_url":"https://api.github.com/users/wkcn/repos","events_url":"https://api.github.com/users/wkcn/events{/privacy}","received_events_url":"https://api.github.com/users/wkcn/received_events","type":"User","site_admin":false},"created_at":"2018-03-02T02:13:40Z","updated_at":"2018-03-02T02:13:40Z","author_association":"NONE","body":"I wrote a cpu/gpu consistency test for Proposal and MultiProposal.\r\nI found there is difference between the CPU output and the GPU output for mx.nd.contrib.Proposal.\r\n\r\nIt seems that the index order of Nonmaximum may be different.\r\nAnd another problem is that it may need to add the condition `num_to_keep < rpn_post_nms_top_n`\r\nhttps://github.com/apache/incubator-mxnet/blob/master/src/operator/contrib/proposal.cu#L341\r\nreference: https://github.com/apache/incubator-mxnet/blob/master/src/operator/contrib/proposal.cc#L235\r\n\r\nHere is the cpu/gpu consistency test.\r\n```python\r\nimport mxnet as mx\r\nimport numpy as np\r\n\r\n# @with_seed()\r\ndef test_multi_proposal_op():\r\n    # paramters\r\n    feature_stride = 16\r\n    scales = (8, 16, 32)\r\n    ratios = (0.5, 1, 2)\r\n    rpn_pre_nms_top_n = 12000\r\n    rpn_post_nms_top_n = 2000\r\n    threshold = 0.7\r\n    rpn_min_size = 16\r\n\r\n    feat_len = 14\r\n    H, W = feat_len, feat_len\r\n    num_anchors = len(scales) * len(ratios)\r\n    count_anchors = H * W * num_anchors\r\n\r\n    def get_new_data(batch_size, ctx):\r\n        '''\r\n        cls_prob: (batch_size, 2 * num_anchors, H, W)\r\n        bbox_pred: (batch_size, 4 * num_anchors, H, W)\r\n        im_info: (batch_size, 3)\r\n        '''\r\n\r\n        cls_prob = mx.nd.empty((batch_size, 2 * num_anchors, H, W), dtype = np.float32, ctx = ctx)\r\n        bbox_pred = mx.nd.empty((batch_size, 4 * num_anchors, H, W), dtype = np.float32, ctx = ctx)\r\n        im_info = mx.nd.empty((batch_size, 3), dtype = np.float32, ctx = ctx)\r\n\r\n        cls_prob = mx.nd.array(np.random.random(cls_prob.shape), ctx = ctx)\r\n        bbox_pred = mx.nd.array(np.random.random(bbox_pred.shape), ctx = ctx)\r\n\r\n        for i in range(batch_size):\r\n            im_size = np.random.randint(100, feat_len * feature_stride, size = (2,))\r\n            im_scale = np.random.randint(70, 100) / 100.0\r\n            im_info[i, :] = [im_size[0], im_size[1], im_scale]\r\n        return cls_prob, bbox_pred, im_info\r\n\r\n    def check_proposal_consistency(op, batch_size):\r\n        '''\r\n            op is mx.nd.contrib.Proposal or mx.nd.contrib.MultiProposal\r\n        '''\r\n        cls_prob, bbox_pred, im_info = get_new_data(batch_size, mx.cpu(0))\r\n        rois_cpu, score_cpu = op(\r\n                cls_score = cls_prob,\r\n                bbox_pred = bbox_pred,\r\n                im_info = im_info,\r\n                feature_stride = feature_stride,\r\n                scales = scales,\r\n                ratios = ratios,\r\n                rpn_pre_nms_top_n = rpn_pre_nms_top_n,\r\n                rpn_post_nms_top_n = rpn_post_nms_top_n,\r\n                threshold = threshold,\r\n                rpn_min_size = rpn_min_size, output_score = True)\r\n\r\n        gpu_ctx = mx.gpu(0)\r\n\r\n        # copy data to gpu from cpu\r\n        cls_prob_gpu = cls_prob.as_in_context(gpu_ctx)\r\n        bbox_pred_gpu = bbox_pred.as_in_context(gpu_ctx)\r\n        im_info_gpu = im_info.as_in_context(gpu_ctx)\r\n\r\n        rois_gpu, score_gpu = op(\r\n                cls_score = cls_prob_gpu,\r\n                bbox_pred = bbox_pred_gpu,\r\n                im_info = im_info_gpu,\r\n                feature_stride = feature_stride,\r\n                scales = scales,\r\n                ratios = ratios,\r\n                rpn_pre_nms_top_n = rpn_pre_nms_top_n,\r\n                rpn_post_nms_top_n = rpn_post_nms_top_n,\r\n                threshold = threshold,\r\n                rpn_min_size = rpn_min_size, output_score = True)\r\n\r\n        print (rois_cpu.asnumpy(), rois_gpu.asnumpy())\r\n        assert np.allclose(rois_cpu.asnumpy(), rois_gpu.asnumpy())\r\n        assert np.allclose(score_cpu.asnumpy(), score_gpu.asnumpy())\r\n\r\n    check_proposal_consistency(mx.nd.contrib.Proposal, 1)\r\n    check_proposal_consistency(mx.nd.contrib.MultiProposal, 20)\r\n\r\ntest_multi_proposal_op()\r\nprint (\"test ok\")\r\n```\r\n"}}
{"actor":{"display_login":"wkcn"}}